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Chapter 1 



Overview and Summaries 
of Value-f or-money Audits 
and Reviews 



Overview 




MAKING TOUGH FISCAL CHOICES 

Ontario is not alone in facing some tough chal- 
lenges to get its fiscal house in order. One of the 
impacts of the recent global recession is that 
Ontario, like many other jurisdictions, has been 
spending far more than it has been collecting in 
revenues. And just like a household that spends sig- 
nificantly more than it earns by borrowing the extra 
money, this practice can only be continued for so 
long. The Ontario government recognizes this, and, 
as the Minister of Finance said in the 2011 Budget, 
"To overcome this challenge, the government must 
renew its focus on deficit reduction." 

Given the challenging fiscal times, making 
significant inroads in reducing the deficit will be no 
easy matter. Consequently, in many of this year's 
value-for-money audits, we paid particular atten- 
tion to looking for efficiencies and cost savings. Our 
observations in this regard are summarized in the 
following subsections. 

Criminal Prosecutions 

Crown attorneys in the Criminal Law Division of the 
Ministry of the Attorney General prosecute criminal 
charges laid by police forces across Ontario. The 
number of Crown attorneys employed has more 



than doubled over the last two decades, even 
though the number of criminal charges disposed 
of each year has remained relatively constant. 
Factoring in the increasing complexity of today's 
legal environment, the Division needs to determine 
how many Crown attorneys there should be at each 
local office and should make much better use of the 
information it has on the relative workloads, effi- 
ciency and effectiveness of its Crown attorneys. 

Diabetes Management Strategy 

The number of people with diabetes in Ontario 
has more than doubled from 546,000 in 2000 
to 1.2 million in 2010 and is projected to grow 
to 1.9 million by 2020. Type 1 diabetes, which 
accounts for 10% of diabetes cases, is not prevent- 
able; however, Type 2 diabetes, which accounts for 
the other 90%, is largely preventable with lifestyle 
changes that include healthier eating and exercise. 
In 2008, the Ministry of Health and Long-Term Care 
implemented a four-year, $741 -million diabetes 
strategy. We found that, to date, the results have 
been mixed, in that while the availability of care for 
people with diabetes has definitely improved, many 
services were underused and, in some cases, dupli- 
cated. As well, we noted that 97% of the funding 
was earmarked for treating people who already had 
diabetes, with only 3% being allocated for preven- 
tion initiatives. 
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Independent Health Facilities 

Ontario has about 800 health facilities that are 
independently owned and operated, with about 
50% being owned or controlled by physicians. 
These facilities provide primarily diagnostic servi- 
ces, such as x-rays, ultrasounds and CT scans. We 
suggested the Ministry review facility billings for 
unusual billing patterns and review whether the 
facility fees being paid are appropriate in relation 
to the actual costs of providing the services. We 
also noted that in 2009, the Canadian Association 
of Radiologists observed that as many as 30% of CT 
scans and other imaging procedures across Canada 
contribute no useful information or are inappropri- 
ate. The Ministry of Health and Long-Term Care's 
own estimate was that about 20% of independent 
health facility fee billings are likely inappropriate. 

Long-Term-Care Home Placement Process 

Many people are in hospital longer than medically 
necessary waiting for a long-term-care bed, even 
though this is detrimental to their health, is more 
expensive than community-based alternatives such 
as home care or placement in a long-term-care 
home, and limits the availability of hospital beds 
for other patients with more complex health-care 
needs. However, 19% of people waiting in hospital 
for a long-term-care bed had applied to only one 
long-term-care home, even though their selected 
home may have a long wait list. 

Metrolinx-Regional Transportation 
Planning 

Metrolinx is an agency of the Ontario government 
that is responsible for implementing an integrated 
transportation system in the greater Toronto and 
Hamilton area over the next 25 years at an esti- 
mated cost of $50 billion. Our review of several 
significant projects costing hundreds of millions of 
dollars indicated that Metrolinx must strengthen 
its processes relating to infrastructure investment, 
procurement and cost control. 



Ontario Provincial Police 

Over the last two decades, crime rates across Can- 
ada have declined by more than 40%, and Ontario 
has been part of this trend. As well, the number of 
serious motor vehicle accidents has been trending 
down in recent years and calls for service to the 
Ontario Provincial Police (OPP) have remained 
stable since our last audit in 2005. However, OPP 
expenditures (excluding recoveries for police 
services provided to municipalities) have increased 
27% over the last five years because more officers 
have been hired, compensation increases have 
exceeded inflation and cost-savings opportunities 
we previously identified had not been adequately 
acted upon. We noted that the OPP could reduce its 
operating costs by improving its staff deployment 
practices and better controlling overtime. 

Tax Collection 

Most taxes owing to the province are remitted 
voluntarily, but some are not, and the Ministry of 
Finance's Collections Branch is responsible for col- 
lecting most of these taxes. The Branch estimates 
that, as of March 31, 2012, it may need to write off 
about $1.4 billion of the $2.46 billion in outstand- 
ing taxes that it is responsible for collecting. While 
there have been some improvements to the Branch's 
collections processes in the last few years, we found 
that the Branch was often not taking appropriate 
action on a timely basis and was not using all the 
strategies at its disposal to maximize the amount of 
unpaid taxes it collects. 

Youth Justice Services Program 

The Youth Justice Services program of the Ministry 
of Children and Youth Services provides commun- 
ity and custodial programs and services primarily 
to young people aged 12 to 17 who are either 
awaiting trial or have been found guilty by a court 
of a criminal charge. From the 2005/06 to the 
2010/11 fiscal years, total program expenditures 
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increased by 25%-30%, even though the number 
of youths served increased by only 4%. And in the 
last five years, even though the number of youths 
in Ministry-operated secure facilities decreased by 
37%, the number of ministry youth services officers 
at these facilities increased by 50%. 

ENHANCING SERVICES TO THE PUBLIC 

Providing public services as cost-effectively as pos- 
sible is critical to managing Ontario's current fiscal 
situation. However, ensuring that the public is get- 
ting the best level of service possible for the funding 
being provided is equally important. We therefore 
also made service delivery — the level of service 
provided and the processes followed in offering 
those services — a key focus of some of our audits. 
Our observations in this regard are summarized in 
the following subsections. 

Cancer Screening Programs 

Screening that detects certain types of cancer at an 
early stage can have a major impact on mortality 
rates. Cancer Care Ontario (CCO) is the provincial 
agency responsible for co-ordinating and overseeing 
the provision of such screening services. Our audit 
assessed whether CCO used established clinical 
evidence to decide what types of cancer warranted 
formal screening programs and how effective CCO 
was in achieving high screening participation rates. 
Overall, we found that CCO had implemented a 
number of good processes but was having difficulty 
meeting its participation-rate targets, especially for 
those segments of the population deemed to be at a 
high risk for certain types of cancer. 

Drive Clean Program 

The mandatory Drive Clean vehicle-emissions- 
testing program has been in place since 1999. 
Vehicle emissions have declined significantly since 
then and are no longer among the major domestic 
contributors to smog in Ontario. While this is due 



partly to the Drive Clean program, newer vehicles 
with improved vehicle-emission-control systems 
and cleaner fuel requirements have had a larger 
impact on reducing overall vehicle emissions. We 
encouraged the Ministry to ensure that future 
policymakers are provided with current informa- 
tion on the program's cost-effectiveness and its 
impact on reducing smog, especially outside of 
Ontario's larger cities and in relation to other prov- 
incial and federal government initiatives, to ensure 
the program continues to provide value for the fees 
charged to drivers. 

Education of Aboriginal Students 

Many aboriginal students face challenges — such as 
poverty, substandard housing, poor nutrition and 
minimal employment prospects in their commun- 
ity — that affect their academic achievement. There 
is a significant gap between the percentage of the 
general Ontario population that has graduated 
from high school and the percentage of aboriginal 
adults that has done so. Six years ago, the Ministry 
of Education identified closing this gap as a ministry 
priority and developed a good framework to guide 
the development of support programs and to track 
its progress in reducing the gap. However, we found 
that the Ministry has not adequately overseen local 
school boards' implementation of the framework 
and has not formally assessed whether any progress 
has been made in closing the academic achievement 
gap. Information we examined suggests that little 
substantive progress has been made to date. 

University Undergraduate Teaching Quality 

From the perspective of the Ministry of Training, 
Colleges and Universities, a university's most 
important role is to do a good job teaching its stu- 
dents and preparing them for the future workforce. 
We believe that students, their parents and the 
public would agree. Accordingly, we reviewed the 
processes that three universities had in place to 
periodically assess and report on the performance 
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of those teaching undergraduate students. We also 
surveyed Ontario's other 17 universities on some 
of their practices in this area. As part of these prac- 
tices, all Ontario universities, including the three 
visited, reported that they had formal processes in 
place to enable undergraduate students to evalu- 
ate each course they took. However, at least at the 
universities we visited, little aggregate analysis of 
these student evaluations was being done, and only 
about one-quarter of the universities responding to 
our survey indicated they make any summarized 
results of these evaluations available to students to 
help them select their courses. 

FINANCIAL OVERSIGHT AND 
ACCOUNTABILITY 

Annual Reporting of Financial Results 

While our value-for-money audit work usually 
draws the most attention from the Legislature, 
the public and the media, financial audit work is 
also a crucial responsibility of our Office. Annual 
reporting of financial results, whether those of 
the province or those of the province's many 
Crown agencies and corporations, has long been 
considered an essential component of the govern- 
ment's financial accountability to Ontarians. But 
the accountability loop is not closed until the Legis- 
lature and the public are assured that the reported 
results are, in fact, fairly stated. 

Accordingly, our Office audits the financial 
statements of the province and many of its Crown 
agencies, such as the LCBO, the Ontario Securities 
Commission, Ontario Place and many others. In 
some cases, such as with the financial statements of 
Cancer Care Ontario and the Public Guardian and 
Trustee, we engage private-sector auditors to do 
this work on our behalf. 

I am pleased to report that for the 19th straight 
year, the Office was able to provide the Legislature 
and the public with the assurance that the consoli- 
dated financial statements of the province — the 
largest audited entity in Ontario — were fairly 



presented in accordance with generally accepted 
accounting principles as established by the Can- 
adian Institute of Chartered Accountants. I can also 
report that we concluded that the financial state- 
ments of the many Crown agencies that we audited 
this year were also fairly presented. 

Legislative Estimates Review Process 

Given that the government spends over $120 billion 
a year on public services and that we last reviewed 
the estimates review process more than 15 years 
ago, we decided to revisit this area in 2012. 

We researched the estimates review practices 
of other jurisdictions with similar parliamentary 
systems, such as the United Kingdom and Australia, 
and interviewed nine currently serving MPPs — 
three from each of the major political parties — who 
have sat on the Standing Committee on Estimates. 
We also met with three retired long-serving MPPs — 
one from each of the parties — who together had 
served a combined total of 80 years as elected MPPs 
in Ontario's Legislature. 

The consensus of both the current and former 
MPPs was that the process was still not that 
effective as a means of scrutinizing proposed 
government expenditure plans. However, almost 
everyone we spoke to said that it was still a good 
accountability mechanism in that it was the only 
real opportunity outside of question period for 
questioning a Minister on the policies and expendi- 
tures of his or her ministry. 

OTHER WORK 

Advertising Review and Approval 

The Government Advertising Act, 2004, requires 
that our Office review most proposed government 
advertising to ensure that it is not partisan in nature. 
We reviewed 565 individual advertising items this 
year. The results of our work in this area can be 
found in Chapter 5. This year, for the first time, the 
spending on Internet advertising exceeded spending 
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on television advertising (excluding ad production 
costs) — yet Internet advertising is not covered by 
the Act. 

Special Audits 

Under the Auditor General Act, we perform assign- 
ments as may be required by the Legislature, 
by a resolution of the Standing Committee on 
Public Accounts, or by a minister of the Crown. We 
normally table these audit reports on completion 
rather than including them in our Annual Report. 
This year we completed one special audit report, 
Ornge Air Ambulance and Related Services. We 
began this audit in 2011 and were asked to table it 
on completion. We did this in March 2012. To date, 
this audit report has been the subject of 17 hearings 
before the Standing Committee on Public Accounts. 



of Annual Reports 



This is the 10th Annual Report I have had the 
honour of tabling with the Legislative Assembly. 
In looking back, I think the most fitting adage to 
describe these 10 years is "how time flies." Much 
has happened over the past decade from the per- 
spective of my Office and the work we do on behalf 
of the Legislature and the people of Ontario. 

I thought that it would be useful to provide an 
overview of some of the more significant events 
that have had an impact on the work of the Office 
as the independent watchdog that provides elected 
members and the public with objective information 
on how well the government is taking care of our 
tax dollars. As well, I offer a few forward-looking 
thoughts and some good news. 



VALUE-FOR-MONEY AUDITING IN THE 
BROADER PUBLIC SECTOR 

Undoubtedly, the most significant event for the 
Office in the past decade has been the Legislature's 
decision to expand our mandate in late 2004 to 
allow us to conduct value-for-money audits of 
organizations in the broader public sector. These 
organizations include hospitals, school boards, col- 
leges and universities, long-term-care homes, social 
service agencies such as Children's Aid Societies 
and hundreds of other organizations that provide 
services to the public. In total, about 40% of Ontario 
government expenditures, or about $47 billion, goes 
to these broader-public-sector entities. 

We have been diligent in pursuing this expanded 
mandate by conducting audits across virtually the 
whole spectrum of organizations in the broader 
public sector. This has included everything from 
assessing the detection and prevention of infectious 
diseases in hospitals and long-term-care homes to 
reviewing the teaching of undergraduate students 
at universities; and from reviewing the purchasing 
of goods and services at school boards, colleges and 
the hydro corporations to probing the interrelation- 
ship between patient care at hospital emergency 
departments, discharge of patients from hospitals, 
and the provision of home care and long-term care. 
The focus of this work has always been to provide 
useful information on how well these services are 
being delivered and to point out where we believe 
improvements can be made. 

GOVERNMENT ADVERTISING AND THE 
PRE-ELECTION FISCAL REVIEW 

The Legislature also gave us two more responsibil- 
ities in 2004. The Government Advertising Act, 2004, 
requires the Office to review proposed government 
advertising to ensure that partisan ads are not paid 
for with taxpayer dollars. The Fiscal Transparency 
and Accountability Act, 2004, requires that the 
government produce a fiscal outlook report six 
months before a provincial election, and our Office 
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is required to promptly review this forecast and give 
an opinion on its reasonableness. 

Although few other jurisdictions have imple- 
mented legislation of this nature, I believe these 
initiatives have enhanced government accountabil- 
ity to the people of Ontario. Admittedly, reviewing 
proposed advertisements and giving an opinion on 
forecasts of future events are somewhat unusual 
pursuits for an auditor. However, I fully support the 
continuing role of the Office in these two areas and 
believe that the objective scrutiny that we bring to 
the process enhances the value of our Office to the 
Legislature and the public it serves. 

SPECIAL AUDITS 

Our special assignments have become more 
frequent in recent years, and I like to think this is 
because we have established a reputation for doing 
good work and being objective and independent in 
how we approach our work. 

Some of these special audits have garnered a 
great deal of attention in the Legislature, from 
the media and the public, and from the Standing 
Committee on Public Accounts. A few reports that 
come to mind over the years are the Ornge Air 
Ambulance and Related Services audit tabled last 
March, our 2009 audit of eHealth, our 2007 exam- 
ination of the cost to refurbish nuclear reactors at 
Bruce Power and our 2007 audit of the Ministry of 
Citizenship and Immigration's year-end grants. 

OUR VALUE-F0R-M0NEY TRACK RECORD 

Given our focus on value-for-money auditing, a rea- 
sonable question to ask is whether our own Office 
operates cost-effectively and with due regard for 
value for money. As discussed earlier, our Office's 
responsibilities have expanded significantly since 
2004. However, this in itself does not necessarily 
prove that taxpayers are getting value for money for 
the work done by the Office. An equally important 
consideration is the cost of doing this work. For 
instance, has there been a significant increase in 



staff or in the amount of money the Office is spend- 
ing due to the expansion of its mandate? 

In the 2011/12 fiscal year, our Office cost tax- 
payers $15.2 million, with the bulk of that amount 
going to salaries. As Figure 1 shows, our staffing 
has increased only slightly over the past decade, 
with the only noticeable bump being a small 
increase shortly after 2004 when our mandate was 
expanded in the three different areas discussed 
earlier. We have tried to run a tight fiscal ship so 
that we could undertake the work required by the 
expansion in our mandate without requesting sig- 
nificantly more funding. 

When people ask about our budget, they often 
want to know how the size of our Office compares 
to that of other Auditor General offices in Canada's 
larger provinces. As of March 31, 2012, with a staff 
count of 102 people, we operated with fewer staff 
than the Quebec, British Columbia and Alberta 
audit offices. As different offices have different 
mandates and responsibilities, comparing the 
number of staff in one office to another is not a 
strictly apples-to-apples comparison. However, it 
does indicate that our staffing levels are certainly 
reasonable compared to those of the audit offices 
of the other larger Canadian provinces. 



Figure 1: Number of Staff at the Office of the Auditor 
General, 2002/03-2011/12 

Source of data: Office of the Auditor General of Ontario 
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LOOKING FORWARD 

If one were to flip through the 10 Annual Reports 
that I have tabled over the past decade, one would 
see a myriad of recommendations that are directed 
to improving the operations of government. Often, 
each year's Annual Report contains a theme. These 
have ranged from "spend taxpayer money like it 
was your own" to the "need for better oversight" 
to "deja vu," where I quoted a statement from the 
1970s by Ontario's Committee on Government 
Productivity "that the challenges of the next decade 
will be very different, more complex, and more 
demanding than those which face us today." But 
if I had to pick one theme above all others that I 
believe the public sector must keep in mind going 
forward, it would be that "good decisions require 
good information." 

Time and time again, the work of the Office 
has highlighted instances where decision-makers 
would have benefited from more relevant, reliable 
and timely information. Our concerns in this area 
sometimes resulted from management information 
systems that were not designed to give decision- 
makers the information they needed. In other 
cases, however, the information was there but just 
not being properly used to support the decisions 
being made. Some of the more significant examples 
that come to mind relate to our audits of the costs 
of renewable energy initiatives, new funding mech- 
anisms for physicians, the process for assessing 
property taxes, the operation of the Family Respon- 
sibility Office, funding for social-service agencies, 
and the land and air ambulance programs. 

As I have said in the past, a lack of good infor- 
mation significantly increases the risk that the 
decisions made will not be the best ones — and 
may even be the wrong ones. In the private sector, 
wrong decisions usually just cost more money; in 
the public sector, they not only cost more but they 
also affect the day-to-day lives of all Ontarians. 



THE GOVERNMENT'S FINANCIAL 
STATEMENTS 

Auditors deal with numbers, and therefore this look 
back to 2003 would not be complete without at 
least some reference to accounting. As I mentioned 
earlier in this chapter, the role we play in providing 
assurance and adding credibility by auditing the 
annual financial statements of the province and 
its Crown agencies should not be overlooked, even 
though it is our value-for-money work that tends to 
capture the attention of the Legislature, the public 
and the media. 

For the past 19 years, my predecessor Erik Peters 
and I have been able to give the government's con- 
solidated financial statements a clean bill of health. 
All three major political parties, at one time or 
another during these years, have formed the govern- 
ment, and all deserve credit for accomplishing this. 

However, in the last few years, the government 
has chosen to legislate how certain transactions are 
to be accounted for. This has not so far resulted in 
any significant departure from generally accepted 
accounting principles, but I do have concerns about 
it. I hope that, in future years, this will not put the 
Auditor in the position of concluding that, although 
the accounting complies with legislation, the 
financial statements are not fairly presented under 
generally accepted accounting principles. 

PLAIN-LANGUAGE ACCOUNTING 

I must confess to some discomfort with what I see 
as a trend toward growing complexity in accounting 
principles and the accompanying financial state- 
ment disclosures. It is reaching the point where the 
statements are becoming so long and complicated 
that the average reader may not be able to quickly 
and easily understand what is being reported with 
respect to an entity's financial position and results 
for the year. 

The recent recommendation of the Canadian 
Institute of Chartered Accountants (CICA) that gov- 
ernment business enterprises adopt International 
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Financial Reporting Standards (IFRS) has not, in my 
opinion, made this situation any better. While I can 
understand the rationale behind the CICA's decision 
to pattern Canadian standards after international 
standards, I suspect that it has resulted in the lay 
reader of financial statements prepared under IFRS 
having more difficulty in understanding them. 

For instance, we have been the auditor of the 
LCBO for many years. From a business and account- 
ing perspective, the LCBO is a fairly straightforward 
business. It buys a finished product and sells it in 
retail stores throughout the province. But the adop- 
tion of IFRS has resulted in the LCBO's financial 
statements increasing from 14 pages to 45 pages. 
As well, there can now be two income statements, 
making it more difficult for the reader to figure out 
what the LCBO's bottom-line profit or loss — the 
key piece of information most readers look for — 
actually is. 

From the perspective of the typical reader of 
the LCBO's financial statements, I have a hard time 
concluding that this is progress. 

But I do have some empathy for the CICA given 
the globalization of the world economy. And insofar 
as the CICA is regarded as a respected contribu- 
tor to the development of future IFRS accounting 
pronouncements, I hope it keeps in mind the views 
of a former Chairman of the U.S. Securities and 
Exchange Commission, who said: "The legalese 
and jargon of the past must give way to everyday 
words that communicate complex information 
clearly." Closer to home, I have always remembered 
the words of a long-serving member of Ontario's 
Standing Committee on Public Accounts who made 
the following comment when the Committee was 
discussing how to word a particular report item: 
"Sometimes, just plain English is OK." 

SOME GOOD NEWS 

In my opening remarks at the press conference 
that is held each year on the release of our Annual 
Report, I have always included a "good news" com- 



ponent where I outline some examples of things 
that have gone well or where we saw some good 
initiatives undertaken. If I were to pick three over- 
riding positive things that come to mind in looking 
back over the past 10 Annual Reports, they would 
be the following. 

Implementation of Our Recommendations 

In my first Annual Report to the Legislative Assem- 
bly in 2003, 1 expressed the concern that, all too 
often, little action had been taken to address our 
recommendations. Since that time, however, there 
has been a definite improvement in this area. 
In 2007, 1 reported a favourable trend in imple- 
menting our recommendations, and, as can be seen 
in Chapter 4 of our more recent Annual Reports 
(where we report the results of our follow-ups), 
progress continues to be made in implementing 
our recommendations. 

Two factors have been instrumental in this. 
First, I sense that there is more pressure from the 
"centre" or Cabinet Office to ensure that action is 
being taken on the Auditor's recommendations. My 
discussions with the various secretaries of Cabinet 
Office since 2003 have indicated that implementing 
our recommendations is considered an important 
responsibility of the deputy ministers. 

Second, the Standing Committee on Public 
Accounts has consistently brought pressure to bear 
on ministries, Crown agencies and organizations in 
the broader public sector to implement our recom- 
mendations. Before any hearing, the Committee 
Clerk formally requests a written response from the 
entity at the hearing outlining what actions it has 
taken to address the Auditor's recommendations. 

Admittedly, the fact that we do a formal follow- 
up two years after each audit and report the 
results in every Annual Report has in itself had an 
encouraging impact. However, the roles played by 
the two parties discussed above have undoubtedly 
been a factor in the improvement we have seen in 
this area since 2003. 
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Public-sector Senior Management 

Over the years, I have dealt with dozens of deputy 
ministers, assistant deputy ministers, and Crown 
agency CEOs and CFOs, as well as a cross-section 
of senior management in broader-public-sector 
organizations such as hospitals, universities, social- 
service agencies, long-term-care homes and many 
others. The times when I have come across some- 
one who didn't seem committed to delivering the 
best possible service for the people he or she serves 
have been rare. 

This doesn't mean that senior management 
always makes the best decisions or ensures that 
every decision is made putting taxpayers' interests 
first and foremost. However, my sense has almost 
always been that, as the saying goes, "their hearts 
are in the right place." And while there have cer- 
tainly been exceptions, the senior management I 
have dealt with over the years has generally been 
reasonable and open-minded with respect to our 
observations and recommendations. As well, 
while no one likes a visit from the auditors, we 
have almost always received good co-operation. 
In this regard, I do want to recognize the excellent 
co-operation of organizations in the broader public 
sector that we have audited, most of which we were 
visiting for the first time. 



the fire" if it believed the ministry or agency had 
not done enough to make improvements. 

For the most part, the Committee has also oper- 
ated in a relatively non-partisan manner and has 
worked collegially to encourage improvements 
in operational cost-effectiveness and improve the 
level of service provided to the public. Last fall, 
just before the House adjourned for the October 
2011 election, the Speaker of the Legislature put it 
best: "It has been a pleasure to see how the Public 
Accounts Committee has worked. If only every com- 
mittee in the Legislature could operate in the same 
manner, things might be much different." 

THANKS TO A HARD-WORKING GROUP 

In looking back over the past 10 Annual Reports 
that I have tabled, I have indeed been most fortun- 
ate to have such a cohesive and dedicated team in 
the Office. Year after year, my staff has continued 
to produce top-notch work, both on the value-for- 
money side and in getting our financial audits done 
well and on time. The work we do is not easy, and 
we frequently face obstacles, challenges and tight 
timelines. However, the staff of the Office perse- 
veres and ultimately gets the job done — and done 
right. They deserve our thanks. 
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Standing Committee on Public Accounts 

Ontario has a very diligent and active Public 
Accounts Committee. I have worked closely with 
the Committee since 2003, and its support for the 
Office and the work we do cannot be overestimated. 
From a pragmatic perspective, its support gives 
us clout, especially when it comes to getting our 
recommendations implemented, as just discussed. 
For example, over the years, it has not hesitated 
to re-call a particular ministry or agency and hold 
another hearing to, as the former long-serving 
Committee Chair liked to put it, "hold their feet to 



About two-thirds of the Office's resources are 
devoted to conducting value-for-money audits. 
These audits focus on the delivery of services to 
the public, going beyond "just the numbers." The 
following are the summaries of the 12 value-for- 
money audits and the one review we conducted, all 
of which are presented in Chapter 3. 
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3.01 CANCER SCREENING PROGRAMS 

Cancer Care Ontario (CCO) is a provincial agency 
responsible for coordinating and overseeing cancer 
services in Ontario. CCO directs health-care fund- 
ing to hospitals and other care providers, with 
the aim of delivering quality and timely cancer 
services throughout the province. It is also respon- 
sible for implementing cancer prevention and 
screening programs. 

CCO has 13 regional cancer programs. Regional 
Cancer Centres are responsible for cancer screen- 
ing and treatment services. In the 2011/12 fiscal 
year, CCO had total expenditures of $887 mil- 
lion, $92 million of which was spent on cancer 
screening programs. 

CCO has implemented screening programs for 
breast, colorectal and cervical cancers. We noted 
that CCO used recognized clinical evidence to 
decide what types of cancer warranted formal 
screening programs. 

Each of the three screening programs has as its 
key objective reducing the number of deaths from 
cancer through early detection and treatment. The 
mortality rates from these three types of cancer 
have fallen in Ontario over the past two decades. In 
this regard, Ontario's mortality rates are similar to 
the Canadian averages for these types of cancer. 

Our observations with respect to the three 
screening programs include the following: 
• Both the Ministry of Health and Long-Term 
Care (Ministry), through a $45-million fund- 
ing commitment in 2010, and CCO, through 
recent initiatives, have recognized the need 
to increase screening participation rates, 
especially for people considered to be at 
increased risk for cancer. As of fiscal 2009/10, 
participation in breast cancer and cervical 
cancer screening achieved Ministry targets 
but fell short of CCO's own targets. Colorectal 
cancer screening fell short of both the Min- 
istry's and CCO's targets, and almost half the 
targeted population remained unscreened. 
In total, from 2008 to 2010, only 27% of 



eligible women completed all three cancer- 
screening tests recommended for their age 
group. As well, participation in the screening 
programs appears to have reached a plateau, 
and the CCO is actively looking at ways to 
address this. 

• Though older women were at greater risk of 
dying of cervical cancer, they were screened at 
a much lower rate than younger women. 

• The level of quality assurance measures 
for each of the screening programs varied 
considerably. CCO has a comprehensive qual- 
ity assurance program for the breast cancer 
screening program. However, 20% of screen- 
ings took place outside CCO's program and 
were not subject to the requirements. CCO 
had some quality assurance processes in place 
for the colorectal cancer screening program, 
but none for the cervical cancer program. 

With respect to the wait times at various stages 
of the screening processes for all three types of can- 
cer, we noted the following: 

• Mammography screening wait times for 
women with average risk for breast cancer 
ranged from just over two weeks to IOV2 
months. CCO found that, in its program 
that targets women considered at high risk 
for breast cancer, the wait time for genetic 
assessments for screening eligibility averaged 
84 days. 

• For colorectal screening, almost 30% of cases 
did not have the recommended follow-up 
colonoscopies within the benchmark time 
established by CCO. Our review of hospital 
records found instances where wait times 
were as long as 72 weeks for people with 
family histories and 17 weeks for those with 
positive Fecal Occult Blood Test results. 

• For cervical cancer screening, a recent CCO 
preliminary review showed that the median 
wait time for a colposcopy (a follow-up on an 
abnormal cervical Pap test result) for high- 
grade abnormalities was about three months. 
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3.02 CRIMINAL PROSECUTIONS 

The Criminal Law Division (Division) of the Min- 
istry of the Attorney General (Ministry) prosecutes 
criminal charges on behalf of the Crown before 
provincial courts. The Division receives about 
600,000 new criminal charges each year from more 
than 60 police forces in Ontario. A Crown attorney 
is to prosecute a criminal charge only if it is in the 
public interest to do so and there is a reasonable 
prospect of conviction. 

The Division operates from its head office in 
Toronto, six regional offices and 54 Crown attorney 
offices across the province. The Division's operat- 
ing expenses totalled $256 million in the 2011/12 
fiscal year, 84% of which was spent on staffing. The 
Division employs about 1,500 staff, including about 
950 Crown attorneys. 

The number of Crown attorneys and the 
overall staffing costs for the Division have more 
than doubled since our last audit in 1993. Yet 
the number of criminal charges that Crown 
attorneys dispose of per year has not substantially 
changed— 572,000 in 1992, compared to 576,000 
in 2011. 

Partly as a result of the Charter of Rights and 
Freedoms, many cases are more complex than they 
used to be, so more time is needed to prosecute 
them. Also, more Crown attorneys have been 
assigned to cases involving guns and gangs and 
other dangerous and high-risk offenders. 

However, it is difficult to gauge the actual 
impact of this on prosecutors' workloads because 
the Division makes little use of data to analyze the 
relative workload, efficiency and effectiveness of its 
Crown attorneys. Instead, it relies more on informal 
oversight by senior staff at each of the 54 Crown 
attorney offices. We noted the same issue in 1993. 

We continue to believe the Division would bene- 
fit from having information systems that provide it 
with reliable data on prosecutors' workloads, the 
outcomes of prosecutions, the average time it takes 
to resolve charges, and other key performance indi- 
cators, at the level of both local offices and individ- 



ual Crown attorneys. The Division could also make 
better use of information on court activities that is 
already available, until it completes the develop- 
ment of its own information systems. 

Our other major observations include 
the following: 

• The Division does not formally assess its 
prosecutorial performance. It does not gather 
information on how efficiently charges are 
screened by Crown attorneys before a case 

is prosecuted; how long it takes Crown 
attorneys and staff to prepare cases; whether 
court diversion programs for resolving minor 
criminal charges are used appropriately; 
the number of bail release applications and 
their results; and the outcomes of cases. For 
example, the rates at which some Crown 
attorney offices went to trial were up to 20 
times higher than the rates of other offices. 

• No staffing model has been established to 
determine how many Crown attorneys should 
be at each local office, and there is no bench- 
mark for what a reasonable workload for each 
Crown attorney should be. Workloads varied 
significantly among local offices and between 
regions — 572 charges per Crown attorney at 
one office and 1,726 at another, for example. 

• Of the Division's six regions, the Toronto 
Region disposed of the most charges in total 
in fiscal 2011/12, but it did so at the highest 
cost per charge — $437, compared to the aver- 
age of $268 for the other regions. The Toronto 
Region also disposed of an average of 40% 
fewer charges per Crown attorney than the 
average of other regions. 

• An electronic case-management system, 
originally projected to cost $7.9 million 
and be completed by March 2010, has been 
significantly delayed because of weak project 
management oversight, and the fact that 
insufficient resources have been dedicated to 
the project. 
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3.03 DIABETES MANAGEMENT 
STRATEGY 

Diabetes, which results from the body's partial or 
complete inability to produce and/or properly use 
insulin, is one of the most common chronic diseases 
in Ontario. It can lead to kidney failure, heart 
attack, stroke, amputation and blindness if poorly 
managed or left untreated. Type 1 diabetes, which 
accounts for 10% of cases, is not preventable and 
its cause remains unknown. However, Type 2 dia- 
betes, which accounts for the other 90% of cases, is 
most often preventable with lifestyle changes that 
include healthier eating and exercise. 

The number of people with diabetes in Ontario 
has more than doubled from 546,000 in 2000 to 
1.2 million in 2010, and that number is expected to 
grow to 1.9 million by 2020. People with diabetes 
use the health-care system at about twice the rate 
of the general population, and the cost to Ontario's 
health-care system is expected to grow from 
$4.9 billion in 2010 to $7 billion in 2020. 

In 2008, the Ministry of Health and Long-Term 
Care (Ministry) established a four-year $741 million 
Ontario Diabetes Strategy (Strategy) . The Strategy's 
short-term results have been mixed. The availability 
of diabetes care has definitely improved. However, 
most diabetes service providers set up with Strategy 
funding are underused, and many told us more of 
their funding should go toward preventive services. 
We noted that 97% of the funding was earmarked 
to treat people who already had diabetes, with only 
3% for prevention initiatives. 

Some of our other observations were as follows: 
• eHealth Ontario's efforts to produce an elec- 
tronic Diabetes Registry to allow physicians 
and the Ministry to monitor patient data have 
been problematic. eHealth had been working 
with a private-sector vendor on the Registry, 
but the original completion deadline of April 
2009 was not met, and the proposed release 
date was extended many times. Subsequent 
to our audit fieldwork, the contract with the 
vendor was terminated in September 2012. 



• eHealth and the vendor signed a $46 million 
contract in 2010 that stipulated the vendor 
would be paid only after the Diabetes Registry 
was launched. eHealth has acknowledged 
that this contract traded away much of the 
province's control over the project's design, 
progress and delivery time in exchange for 
price certainty. Although no payment had 
been made to the vendor and the Registry was 
cancelled in September 2012, the Ministry 
and eHealth have incurred about $24.4 mil- 
lion in internal costs related to the Registry 
since 2008/09. 

• There has been considerable duplication and 
overlap in education programs on diabetes. 
The provincial Strategy runs 152 Diabetes 
Education Programs (DEPs), each with one or 
more Diabetes Education Teams consisting of 
a registered nurse, a registered dietician and 
other professionals. However, many hospitals 
and physicians' clinics have set up education 
programs of their own, with funding from 
other sources, leading to service overlaps and 
under-utilization of 90% of the DEPs. 

• The Ministry needs to significantly enhance its 
monitoring of funds used by a not-for-profit 
organization to which it gives $20 million a 
year to manage and fund 47 DEPs in northern 
Ontario and a number of diabetes service pro- 
viders. The organization has paid a consulting 
firm $105,000 since 2009/10 for such services 
as "advice on election strategizing" and 
"developing relationships with relevant polit- 
ical decision-makers." We also found instances 
where staff meal expense claims were not in 
line with government policy. 

• The Ministry has significantly increased the 
number of in-province bariatric surgeries — 
from 245 in 2007/08 to 2,500 in 2011/12— to 
combat Type 2 diabetes in obese people. 
However, this still does not meet the current 
demand and is actually lower than the 2,900 
surgeries done in 2009/10. 



Overview and Summaries of Value-for-money Audits and Reviews 



3.04 DRIVE CLEAN PROGRAM 

The Ministry of the Environment (Ministry) intro- 
duced the mandatory Drive Clean vehicle emissions 
program in 1999 as part of its strategy to reduce 
smog in Ontario. The program identifies vehicles 
whose emission controls are malfunctioning and it 
requires that the owners of such vehicles have them 
repaired. 

The program currently tests vehicles once they 
are seven years old, or those older than one year if 
ownership is to be transferred. Light-duty vehicles 
that were built before 1988 are exempt from the 
program, but, otherwise, all vehicles must pass an 
emissions test for the owner to renew the registra- 
tion or transfer ownership. 

Overall, we found that the Drive Clean program 
has effective procedures in place to ensure that 
vehicles are getting tested and that vehicles whose 
emissions exceed the province's limits are being 
identified for repair. 

On-road vehicle emissions declined significantly 
from 1998 to 2010, and they are no longer among 
the major domestic contributors to smog in 
Ontario. (Half of Ontario's smog comes from pol- 
lutants that originate in the United States.) As well, 
Ministry emissions estimates show that more than 
75% of the reduction in vehicle emissions since the 
Drive Clean program's inception is actually due to 
factors other than the program, including tighter 
manufacturing standards on emission-control 
technologies, federal requirements for cleaner fuel 
and the fact that older vehicles are being retired. 

Some of the other significant issues we noted 
during our audit were as follows: 

• Beginning January 1, 2013, the program is 
to begin using an on-board diagnostic (OBD) 
testing method, which can only test vehicles 
built from 1998 on. As a result, vehicles built 
from 1988 to 1997, which experienced a fail- 
ure rate of 11% to 31% in 2010 when tested 
with a dynamometer, will be tested using 
only the two-speed idle method, which uses 



less stringent emissions limits than either the 
dynamometer or the OBD testing methods. 

• Because vehicle owners are not required to 
incur any repair costs if the repair estimate 
exceeds $450, about 18,000 vehicles were 
not fully repaired in 2011. The average repair 
bill paid by owners of vehicles that received 

a conditional pass was only $255. The most 
commonly diagnosed cause of excessive emis- 
sions in 2010 — a faulty catalytic converter — 
was repaired in only one-third of cases. For 
vehicles that had only partial repairs in 2011, 
the emission readings after the repair were 
actually worse for all pollutants in 25% of the 
vehicles, and worse for at least one of the pol- 
lutants in half of the vehicles. 

• The Ministry outsources six program services, 
including the monitoring of Drive Clean facili- 
ties for non-compliant or fraudulent activities, 
to the private sector. It recently consolidated 
the six separate private-sector service delivery 
contracts into one contract and expects a 40% 
reduction in annual costs. Until recently, the 
Ministry has been diligent in requiring its 
service provider to conduct upwards of 1,400 
covert audits a year. In recent years, these and 
other audit efforts have identified about 3,000 
non-compliance issues annually. However, 
prior to the planned introduction of a new 
compliance program in 2013, the Ministry 
reduced the number of covert audits in 2012 
to a fraction of what it previously required the 
service provider to conduct. 

• Although one of the program's stated goals is 
to maintain a high level of public acceptance, 
the Ministry has not established performance 
targets or attempted to measure whether or 
not this goal has been achieved in more than 
a decade. The only survey to measure public 
support for the Drive Clean program was done 
12 years ago. 
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3.05 EDUCATION OF 
ABORIGINAL STUDENTS 

There are about 242,000 Aboriginal people living 
in Ontario, including 158,000 First Nations people, 
74,000 Metis and 2,000 Inuit. Data from the 2006 
census indicates that only 62% of Aboriginal adults 
in Ontario had graduated from high school, com- 
pared to 78% of the general population — a gap of 
16%. The academic achievement gap is up to 50% 
for young adults aged 20 to 24. In this age group, 
only 39% of the First Nations people living on 
reserves had graduated from high school. 

Many Aboriginal students face challenges that 
affect their academic achievement, including 
poverty, substandard housing and poor nutrition. 
Many live in areas with little prospect of employ- 
ment, something that can affect how seriously they 
take getting an education. 

In 2006, the Ministry of Education (Ministry) 
identified Aboriginal education as a priority, with a 
focus on closing the gap in academic achievement 
between Aboriginal and non-Aboriginal students 
by 2016. It created the Aboriginal Education Office 
(AEO), which collaborates with Aboriginal com- 
munities and organizations, school boards, other 
ministries and the federal government, to coordin- 
ate Aboriginal education initiatives. Since 2006, the 
Ministry has provided $170 million in funding to 
support programs for Aboriginal students. 

In 2007, the Ministry designed a policy frame- 
work to identify Aboriginal students, help develop 
support programs and periodically assess their 
academic progress. However, the Ministry needs to 
more actively oversee the implementation of this 
framework to demonstrate what, if any, progress 
has been made since 2006 in improving achieve- 
ment among Aboriginal students. 

Among our significant observations are 
the following: 

• Five years after the release of the Ontario 
First Nation, Metis and Inuit Education Policy 
Framework, the Ministry had not assessed its 
progress against any of the 10 performance 



measures included in the framework because 
it had not required school boards to evaluate 
and report on the measures. None of the three 
boards we visited had done so. 

• The Ministry has a guide to help school 
boards develop policies for students to 
formally identify themselves as Aboriginal. 
However, at the time of our audit, fewer than 
half of the estimated number of Aboriginal 
students in Ontario had been identified. The 
Ministry and boards need to identify Aborig- 
inal students to better target funding and sup- 
port, and determine their academic progress. 

• The Ministry had not established a baseline 
from which to measure the gap in achieve- 
ment between Aboriginal and non-Aboriginal 
students. Accumulation of credits toward 
graduation is a primary indicator of student 
success. We asked the Ministry for the most 
recent data for Grade 10 credit accumula- 
tion for students who identified themselves 
as Aboriginal. Only 45% of these students 
were on track to graduate from high school, 
compared to 74% of the general Grade 10 
population. This raises the question of the 
Ministry's ability to meet its goal of closing the 
achievement gap by 2016. 

• Although education on reserves is the finan- 
cial responsibility of the federal government, 
many of these students eventually transition 
into the provincial system. Partly because 

of limited per-student funding, on-reserve 
schools have generally not been able to 
provide the quality of education found in 
provincial schools, and studies suggest these 
students may be several grade levels behind 
when they transfer into the public system. Our 
analysis of Education Quality and Account- 
ability Office data found that only half of 
on-reserve students attending provincial 
schools passed the Grade 10 Ontario Sec- 
ondary School Literacy Test in the 2010/11 
school year. 
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3.06 INDEPENDENT HEALTH FACILITIES 

In Ontario, about 800 independent health facilities 
provide primarily diagnostic services — such as 
x-rays, ultrasounds and sleep studies — and about 
25 provide surgery — such as cataract and plastic 
surgery — or dialysis. Patients generally need a 
requisition signed by their physician to receive the 
services, and test results are sent to this physician. 

The facilities are independently owned and 
operated, and more than 97% of them are for- 
profit corporations. The Ministry of Health and 
Long-Term Care (Ministry), which is responsible 
for licensing, funding and coordinating quality 
assurance assessments of these facilities, estimates 
that about half of them are owned or controlled 
by physicians, many of whom are radiologists who 
interpret, for example, x-rays. 

The Ministry pays facility owners a "facility fee" 
for overhead costs such as rent, staffing, supplies 
and equipment. In the 2010/11 fiscal year, the Min- 
istry paid $408 million in facility fees. Total facility 
payments increased by about 4% per year from 
2006/07 to 2010/11, primarily because the volume 
of services increased. As well, the Ministry pays 
physicians a standard "professional fee" for each 
service provided in the facilities. The Ministry does 
not track the total of these specific professional fees. 

Since our 2004 audit, the Ministry has improved 
the oversight of facilities. However, several areas of 
concern still remain. For example, the Ministry gen- 
erally does not allow facilities to relocate to under- 
served areas, even though Ministry data indicates 
that patients in about half of Ontario municipalities 
continue to be underserved for certain diagnostic 
services, including radiology and ultrasound. As 
well, the Ministry has not researched the current 
overhead costs associated with providing the ser- 
vices. These costs may have changed significantly 
because new technology that allows certain tests to 
be done much faster often results in lower overhead 
and staffing expenses. 

Our other significant observations include 
the following: 



• Each facility is paid the same amount for each 
type of service provided, regardless of the 
number of services it performs. Consequently, 
larger facilities in urban areas often benefit 
from economies of scale, since costs like rent 
and reception staff salaries do not increase 
proportionately with the number of services 
performed. Paying slightly higher fees in loca- 
tions with smaller populations and lower fees 
in high-density locations might encourage ser- 
vices in underserved areas without additional 
cost to the Ministry. 

• Although the Ministry estimates that about 
50% of facilities are owned or controlled by 
physicians, it has not analyzed the patterns 
of physicians referring patients to their own 
or related persons' facilities. Further, many 
patients assume they must go to the facility on 
their physician's referral form. 

• In 2009, the Canadian Association of Radiolo- 
gists noted that as many as 30% of CT scans 
and other imaging procedures across Canada 
contribute no useful information or are 
inappropriate. The Ministry's own estimate 
was that about 20% of facility-fee tests are 
likely inappropriate. 

• Unlike hospitals, facilities are assessed by the 
College of Physicians and Surgeons of Ontario 
to help ensure that, among other things, 
diagnostic images are being correctly read 

by the facilities' physicians. However, as of 
March 2012, about 12% of facilities had not 
been assessed within the previous five years. 
Even for assessed facilities, the College asses- 
sors did not review the work of all physicians 
working at those facilities. 

• As of March 2012, the Ministry's X-Ray 
Inspection/Services Unit had not inspected 
almost 60% of facilities as frequently as 
required to ensure that radiation-producing 
equipment, including x-ray equipment, was 
appropriately shielded to prevent excessive 
radiation exposure. 
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3.07 LEGISLATIVE ESTIMATES 
REVIEW PROCESS 

The Ontario government spends about $120 bil- 
lion a year on public services. The government sets 
spending priorities and manages service delivery, 
but it must seek the approval of the Legislature 
each year for its spending plans. 

In recent years, the Ontario budget has been 
presented to the Legislative Assembly in early 
spring. However, the budgetary process starts sev- 
eral months earlier. The government provides min- 
istries with broad spending guidelines that reflect 
its priorities and fiscal outlook. The ministries then 
put forward their proposed spending plans for the 
public services they manage. Ministry plans, once 
reviewed and approved by the Treasury Board and 
Management Board of Cabinet, form the basis for 
the budget, which also reflects the expected rev- 
enues to support the proposed expenditures. 

Ministry spending plans are sub-divided by 
program area and spending activity, and these 
are then summarized in a document titled Esti- 
mates. This document, which must be tabled by 
the government no later than 12 sessional days 
after the budget's release, represents the gov- 
ernment's formal request to the Legislature for 
spending approval. 

In Ontario, the Standing Committee on 
Estimates (Committee) reviews the estimates of 
at least six, but not more than 12, ministries or 
government offices each year. The Committee cur- 
rently comprises members of the provincial Parlia- 
ment (MPPs) from the three political parties that 
have elected members. During the review process, 
ministers and senior staff of the ministries appear 
before the Committee to explain their estimates 
and answer questions. When the Committee com- 
pletes its review, it reports back to the Legislature. 

MPPs may also debate the Estimates in the 
Legislature. After the Legislature approves them, 
they constitute the legal spending authority for 
the government. 



We last reviewed the legislative estimates review 
process in 1995. As we did in 1995, we again inter- 
viewed three MPPs from each party who had served 
on the Estimates Committee. We supplemented 
our interviews with current MPPs by meeting with 
three former, long-serving MPPs, who, in total, 
had more than 80 years of experience as elected 
members, to get their historical perspective on the 
legislative estimates review process. The consensus 
of almost all of the MPPs we interviewed echoed 
the comments members made in 1995 — that is, 
that the process is still not very effective in provid- 
ing meaningful scrutiny of government spending 
plans. However, members made it clear that the 
hearings of the Committee are worthwhile because 
they provide the only real opportunity to directly 
question ministers outside of the Legislature's 
question period. 

Our research did not identify any Westminster- 
style parliaments that had truly effective estimates 
review processes. Many are wrestling with the same 
concerns expressed by Ontario MPPs. Indeed, a 
2005 International Monetary Fund study noted that 
Westminster-style legislatures in general have very 
limited budgetary oversight, and ranked the parlia- 
ments of Canada, Australia, the United Kingdom 
and New Zealand as having the weakest budgetary 
oversight processes of any Western countries. 

We offered these ideas for making more effect- 
ive use of the limited time committee members 
have available: 

• Consider a short in camera estimates briefing 
for members in advance of the formal hear- 
ings for each ministry. 

• Consider requesting that ministry briefing 
books combine information on spending plans 
with past and current performance reporting 
to provide a better basis for legislators to 
assess what results are being achieved and 
planned for proposed expenditures. 

• Consider selecting a few specific ministry 
programs for a more in-depth review by the 
Committee, perhaps with only senior ministry 
officials in attendance. 
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3.08 LONG-TERM-CARE HOME 
PLACEMENT PROCESS 

Long-term-care homes (LTC homes) provide care, 
services and accommodation to people who need 
to have 24-hour nursing care available, supervision 
in a secure setting or frequent assistance with activ- 
ities of daily living. 

The Long-Term Care Homes Act (Act) authorizes 
the province's 14 Community Care Access Centres 
(CCACs) to determine eligibility for LTC home 
admission, prioritize eligible people on wait lists 
and arrange placement when a bed becomes avail- 
able. In fiscal 2011/12, CCACs placed more than 
25,000 people, 85% of whom were 75 or older, in 
Ontario's 640 LTC homes. The 76,000 long-term- 
care beds in these homes are 97% occupied. Each 
CCAC reports to one of the province's 14 Local 
Health Integration Networks (LHINs) . The Ministry 
of Health and Long-Term Care (Ministry), to which 
the LHINs are accountable, is responsible for ensur- 
ing that CCACs comply with the Act's LTC home 
placement provisions. 

Since 2005, the number of Ontarians aged 75 
and over has increased by more than 20%, which is 
undoubtedly one reason why the median amount 
of time people wait for an LTC home bed has almost 
tripled — from 36 days in the 2004/05 fiscal year 
to 98 days in 2011/12. Although wait times have 
decreased somewhat since July 2010, when tighter 
eligibility criteria in the Act took effect, Ontario's 
population of those aged 75 and up is expected 
to grow by almost 30% from 2012 to 2021 and to 
further increase beginning in 2021 when the baby 
boomers start to turn 75, likely creating additional 
demand for long-term care. 

Many factors that affect wait times for place- 
ment are out of the control of CCACs. For instance, 
the Ministry is responsible for how many LTC home 
beds are available. As well, people are allowed to 
select the LTC homes they are willing to be placed 
in, and LTC homes may reject applications. 

The Ministry has recognized that, given 
Ontario's aging population, it is critical that 
alternatives be developed to long-term care. CCACs 



use a standardized process to determine client 
eligibility, including considering alternatives to 
long-term care. However, more needs to be done to 
ensure that crisis cases are prioritized consistently. 

Overall, the three CCACs we visited were man- 
aging various areas of their LTC home placement 
process well, but all had areas that needed improve- 
ment. Our observations included the following: 

• 19% of people waiting in hospital for an LTC 
home bed had applied to only one LTC home, 
even though the selected home may have 

a long waiting list. It has been shown that 
remaining in hospital longer than is medic- 
ally necessary is detrimental to a person's 
health, is more costly than community-based 
care alternatives and takes up beds that are 
needed by other patients. 

• March 2012 LTC home wait-list data indicated 
that crisis clients had waited a median of 94 
days up to that point; moderate-needs clients 
had waited 10 to 14 months; and most other 
eligible clients had waited years. During the 
2011/12 fiscal year, 15% of clients died before 
receiving LTC home accommodation. 

• While 36% of clients were placed in their 
first choice of homes, others accepted an 
alternative LTC home but stayed on their 
preferred home's wait list. In March 2012, 
40% of people on wait lists for a particular 
home resided in another home. Because crisis 
clients get priority, non-crisis clients may find 
it difficult to access the more popular homes. 

• Applicants in some areas of the province get 
into LTC homes more quickly than others. 
At one CCAC, 90% of clients were placed 
within 317 days, whereas at another, it took 
1,100 days. 

• Clients able to pay for private or semi-private 
rooms are generally placed more quickly 
because homes can have up to 60% of their 
beds in such rooms, but only 40% of people 
apply for them. 

• The CCACs we visited did not periodically 
review whether the highest priority clients 
were offered the first available beds. 
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3.09 METROLI NX— REGIONAL 
TRANSPORTATION PLANNING 

Metrolinx, an agency of the Ontario government, 
was established to provide leadership in the 
coordination, planning, financing and development 
of an integrated transportation network in the 
Greater Toronto and Hamilton Area (GTHA) . 

In November 2008, Metrolinx formally adopted 
its Regional Transportation Plan (RTP), which sets 
out the priorities, policies and programs for imple- 
menting a GTHA transportation system over 25 
years at an estimated cost of $50 billion. 

The GTHA suffers from congested roads and 
highways, and public transit systems are increas- 
ingly unable to meet the needs of a growing 
population. The creation of a Crown agency such 
as Metrolinx is a reasonable strategy toward imple- 
menting an integrated transportation network in 
the GTHA. Other jurisdictions that have faced this 
problem have used similar stand-alone agencies to 
coordinate regional transportation planning. 

To deliver on its goals, Metrolinx must, among 
other things, consult with stakeholders and make 
sure that individual projects contribute to a seam- 
less and efficient transportation network; that each 
project has a sound cost-benefit analysis; and that 
risks are managed and projects are delivered cost- 
effectively and on time. 

However, our review of several significant 
projects in the early stages of the RTP identified a 
number of issues that must be addressed by Metro- 
linx if it is to meet these best practices, including 
the following: 

• We believe that Metrolinx's initial assump- 
tions about projected annual ridership on 
the Air Rail Link (ARL) between Union Sta- 
tion and Pearson Airport may be optimistic. 
Although a final decision has not been made 
on whether the ARL must recover its annual 
operating costs and any of its capital construc- 
tion costs, operating it on a break-even basis, 
if that is indeed the objective, may not be 
feasible. Metrolinx itself conducted a market 
assessment that suggested ARL ridership may 



not meet its initial assumptions given the esti- 
mated fare level. 

• The two major projects related to the revital- 
ization of Union Station have experienced 
significant cost increases over their initial 
estimates. The cost of restoring the train 
shed could reach $270 million — 25% more 
than Metrolinx's initial estimate. The cost of 
replacing the switches in the Union Station 
Rail Corridor could be more than twice the 
$38 million on the original purchase order. 

As well, we found a number of issues with the 
Presto fare-card system that Metrolinx has deemed 
key to implementing a region- wide integrated 
transit fare system. Specifically: 

• Because fares for GTHA transit systems are 
not themselves integrated, the Presto card has 
not yet facilitated fare integration. 

• Along with Ottawa's transit body, the Toronto 
Transit Commission has only now condition- 
ally approved the adoption of Presto. However, 
to meet the requirements of Toronto and 
Ottawa, Presto Next Generation (PNG) is 
being developed at an anticipated cost of 
$498 million. The total cost of developing 

the original Presto system and PNG may well 
be more than $700 million, placing it among 
the more expensive fare-card systems in the 
world. Instead of competitively tendering the 
development of PNG, Metrolinx decided to 
develop it through open-ended change orders 
under the existing vendor's contract. We 
believe tendering would have informed Metro- 
linx of potential new developers and, possibly, 
of more cost-effective technology solutions. 

• Presto has been in service for about two years, 
but its overall usage among participating 
GTHA transit systems was only about 18% as 
of March 31, 2012. In the 905 region, seven 
of eight municipal transit agencies use Presto, 
but its overall usage on those systems is only 
6%. These transit agencies cannot eliminate 
their old fare systems in favour of Presto 
because of the card's limitations. 
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3.10 ONTARIO PROVINCIAL POLICE 

The Ontario Provincial Police (OPP) provides front- 
line police services in areas that do not have their 
own police forces, patrols provincial highways, and 
conducts investigations into complex criminal cases 
and organized crime. It also offers policing services, 
under contract, to municipalities that request them 
and provides emergency and other support services 
to all communities in the province. 

The OPP employs about 6,300 police officers 
and 2,300 civilian employees. It operates 78 detach- 
ments and 87 satellite police stations. 

OPP operating expenditures totalled $979 mil- 
lion in the 2011/12 fiscal year, with staffing costs 
making up 87% of that amount. The OPP provides 
municipal policing services to 322 municipalities on 
a cost-recovery basis, as well as to 19 First Nations 
communities, and was reimbursed $362 million. 

Over the last two decades, crime rates across 
Canada have declined by more than 40%, and 
Ontario has been part of this trend. Since our last 
audit of the OPP in 2004/05, crime rates reported 
by the OPP have decreased 10%, and serious motor 
vehicle accidents have also been trending down, 
with both fatalities and injuries decreasing. Over 
the last five years, the number of calls for service 
the OPP has responded to or initiated has remained 
relatively stable. 

However, OPP expenditures net of recoveries 
from municipalities have increased by 27% over the 
last five years. Most of the increase has occurred 
because more officers have been hired and staff 
have received higher compensation. We found that 
many other large police forces in Canada have had 
similar expenditure increases, notwithstanding 
the declining rates of crime and serious motor 
vehicle accidents. 

We found in our current audit that many of the 
issues we reported on in 2005 continue to exist. 
Our major observations include the following: 



• We found that officers face significantly differ- 
ent workloads depending on where they are 
assigned, with some officers handling 54% to 
137% more calls than officers in other detach- 
ments. The reason for this maybe a staffing 
model that is almost 30 years old and that is 
used to deploy only about 45% of the 2,800 
front-line officers. 

• In 2005, the OPP told us it was working 
with the RCMP on a new officer-deployment 
computer model. The OPP has since claimed it 
uses this new model, but it does not. In March 
2012, the OPP's existing model calculated that 
the force needed 500 more front-line officers, 
whereas the new model calculated it needed 
50 fewer officers. 

• OPP management had little control over shift 
scheduling at detachments, and almost all offi- 
cers choose to work 12-hour shifts. This results 
in overstaffing during slow early-morning 
hours; addressing this could result in savings 
in the range of $5 million to $10 million. 

• OPP officers are among the highest compen- 
sated officers in Canada. Officers and civil- 
ians receive certain benefits to which other 
members of the Ontario public service are not 
entitled, including significantly better pen- 
sion benefits and other allowances. 

• Although the OPP had lowered its overtime 
costs for 2004/05 by 10% to $33 million, 
overtime costs have increased by 60% to 
$53 million since then. 

• The overall cost of OPP services for municipal- 
ities from 2007 to 2011 increased an average 
of 29% for those with contracts and 19% for 
those without — up to three times the annual 
inflation rate. While municipal officials told 
us that they were very satisfied with the OPP 
services they received, they expressed concern 
about these cost increases. 
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3.11 TAX COLLECTION 

The Ministry of Finance (Ministry), through its Col- 
lections Branch (Branch), is responsible for the col- 
lection of a significant portion of the unpaid taxes 
owed to the province. To collect unpaid taxes, the 
Branch sends notices by mail, contacts taxpayers 
by phone and sometimes visits in person. If taxes 
remain unpaid, collectors can use garnishments, 
register liens, or obtain warrants for the seizure and 
sale of taxpayers' property. 

As of March 31, 2012, about 90% of the tax the 
Branch was responsible for collecting related to 
Corporations Tax and Retail Sales Tax. The Canada 
Revenue Agency (CRA), which is responsible for 
collecting personal income tax on behalf of the 
province, also began administering Corporations 
Tax on behalf of the province in January 2009. 
Similarly, the CRA administers the Harmonized 
Sales Tax, which replaced the provincial Retail 
Sales Tax in July 2010. As a result, about 75% of 
the Branch's 400 staff members were transferred 
to the CRA in March 2012. However, the Ministry 
is still responsible for collecting Corporations Tax 
and Retail Sales Tax amounts owing prior to their 
transfer to the CRA. The Ministry expects that by 
2014 it will have wound down most of its work on 
collecting these amounts. 

In the 2011 Ontario Budget, the government 
proposed centralizing the collection of all gov- 
ernment non-tax revenue within the Ministry 
of Finance. Under this proposal, the Collections 
Branch would also collect non-tax revenue on 
behalf of other provincial ministries. 

Over the last five years, the Branch collected 
about $6 billion of the approximately $330 billion 
in taxation revenue generated by the province, 
while the remaining amount was generally remit- 
ted voluntarily. As of March 31, 2012, the Branch 
expected it may need to write off up to $1.4 billion 
of the $2.46 billion in taxes owing that it was 
responsible for collecting. The write-off would be 
mostly from older accounts that have accumulated 
for years and that have been expensed in the 
accounts of the province. 



The Branch has strengthened its collections 
process in recent years by improving how it priori- 
tizes accounts and by developing guidelines to help 
collectors carry out and document their work. How- 
ever, in a number of cases we reviewed, we found 
that collection actions were often not taken soon 
enough, and that enforcement tools available to 
collectors were not used to their full extent. Some 
of our significant observations are as follows: 

• Research has shown that the probability of 
full collection on a delinquent account drops 
dramatically as time passes. We found that 
once an account entered collections, it took 
an average of seven months for collectors to 
attempt to reach the taxpayer by phone. We 
also noted that in more than two-thirds of the 
cases in our sample, there was at least one 
instance where no collection action was taken 
for six months or more. 

• Visiting a taxpayer's premises increases the 
likelihood of collecting what is owed. Field 
visits were not made in a number of accounts 
we reviewed, although we felt they were 
warranted. For example, the Branch tried 
unsuccessfully for nearly two years to reach by 
phone a taxpayer who owed $100,000 in sales 
tax and had broken a payment arrangement, 
but it made no field visit. 

• The Branch appropriately registered liens 
and warrants on properties, but, in a number 
of cases that we reviewed, it then failed to 
enforce the liens and warrants for the seizure 
and sale of those properties. 

• With the Branch losing 75% of its workforce 
because of the transfer of responsibilities to 
the CRA, many collectors' caseloads doubled 
or even tripled. This could result in even more 
write-offs than are expected. The Branch 
received approval in fiscal 2009/10 to hire 
temporary employees to compensate for the 
eventual loss of personnel, but at the time of 
our audit, the Branch had not fully evaluated 
its post-transfer needs and, as a result, no 
additional staff had been hired. 
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3.12 UNIVERSITY UNDERGRADUATE 
TEACHING QUALITY 

In 2010/11, Ontario's 20 publicly assisted universi- 
ties had the equivalent of about 390,000 full-time 
students eligible for provincial funding. These 
universities employed about 15,000 full-time fac- 
ulty, including tenure-stream staff with teaching 
and research responsibilities, teaching staff with 
no research responsibilities and part-time sessional 
instructors under contract. 

The Ministry of Training, Colleges and Universi- 
ties (Ministry) expects that 70% of all new jobs will 
require education and training beyond the high 
school level, and its goal is to have 70% of Ontar- 
ians attain post-secondary credentials by 2020. 

From the Ministry's perspective, a university's 
most important mandate is that it does a good job 
of teaching its students and preparing them for the 
future workforce. We believe students, their parents 
and the public would agree. 

The deans and faculty or department heads we 
spoke to at the three universities we visited told 
us that it is not easy to quantify and assess under- 
graduate teaching quality. Nevertheless, most felt 
that measures could be developed to offer insight 
into teaching quality. 

Although neither the Ministry nor the universi- 
ties we visited were formally assessing or reporting 
on teaching performance on a regular basis, we 
found information was available that could be 
used to do so. For instance, all Ontario universities 
encourage students to complete formal evaluations 
of each course they take. However, we found that 
little aggregate analysis of the student evaluations 
was done at the universities we visited. Only about 
a quarter of Ontario's universities indicated that 
they make the summarized results of these evalua- 
tions available to students to help them choose 
their courses. 

All three of the universities we visited had 
put some processes in place to improve teaching 
quality, including establishing teaching centres 
and giving consideration to teaching performance 



when making decisions on promotions and tenure. 
However, we feel universities need to better ensure 
that teaching quality is valued, encouraged and 
rewarded. Our key observations are as follows: 

• A number of faculty told us their annual 
performance appraisals did not provide them 
with appropriate feedback on teaching per- 
formance. We noted examples where student 
evaluations had been critical of teaching 
performance, but there was no evidence that 
specific guidance was provided or that faculty 
members had sought assistance to improve 
their teaching skills. None of the universities 
we visited required that written performance 
appraisals be provided to sessional instruct- 
ors, even though these people accounted for 
10% to 24% of full-time-equivalent staff. 

• Ontario universities in general do not require 
faculty members to have formal training 

in teaching. Records at the teaching and 
learning centres of two of the universities 
we visited showed that faculty attendance at 
teaching workshops averaged less than one 
hour per instructor per year. At one university, 
student course evaluation results showed the 
education faculty consistently outperformed 
other faculties in overall teacher effectiveness 
ratings. Interestingly, we were told that virtu- 
ally all members of this faculty had formal 
training in teaching methods. 

• The Ministry is making progress toward 
achieving its goal of having 70% of Ontario's 
population hold post-secondary credentials by 
2020. However, two years after graduation, 
only 65% of graduates surveyed by the Min- 
istry were employed full-time in a job that was 
related to the skills acquired in their studies. 
The Ontario Undergraduate Student Alliance 
indicated to us that students would find infor- 
mation on graduate employment outcomes 
beneficial in choosing their university and 
program of study. 
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3.13 YOUTH JUSTICE 
SERVICES PROGRAM 

The Ministry of Children and Youth Services (Min- 
istry) provides community and custodial programs 
and services to Ontario youths aged 12 to 17 who 
are primarily awaiting trial after being charged 
with a crime, or who have been found guilty by a 
court. The Youth Justice Services program aims to 
reduce the incidence of reoffending and to contrib- 
ute to community safety, largely through rehabilita- 
tive programming. 

During fiscal 2011/12, the average daily popula- 
tion in Ontario's youth justice system was about 
9,200 — 8,600 under community supervision and 
600 in youth custody/detention facilities (200 in 
open facilities and 400 in secure facilities) . 

In 2011/12, the Ministry spent $370 million 
on the Youth Justice Services program, including 
$168 million in transfer payments to approximately 
200 community-based agencies. The federal gov- 
ernment contributed $67 million toward these costs 
under various cost-sharing agreements. 

Like many other jurisdictions, the program 
has undergone a shift in philosophy over the last 
decade, from an incarceration-based approach to a 
community-based rehabilitation approach. 

From the 2005/06 to the 2010/11 fiscal years, 
total program expenditures in the Youth Justice 
Services program increased by 25%-30%, while 
the number of youths served increased by only 
5%. As well, ministry operating costs grew at a 
much faster rate than funding to transfer-payment 
agencies, even though the agencies have had to 
increase the number of programs and services they 
provide because of the shift to community-based 
rehabilitation. 

Our other observations included the following: 
• The growth in direct operating costs is primar- 
ily due to an increase in employee costs. Over 
the five-year period ending 2010/11, the num- 
ber of full-time employees in all youth justice 
program areas increased substantially, with 
the exception of probation offices. More than 
60% of all full-time ministry staff in the Youth 



Justice Services program were working in 
Ministry-operated secure facilities. Although 
the average daily youth population in these 
facilities decreased by 37% from 2006/07 to 
2010/11, the number of full-time youth servi- 
ces officers increased by 50%. 

• In fiscal 2010/11, on average, about 50% of 
the beds in custody facilities were occupied. 
Over the years, the Ministry has tried to 
improve the utilization rate by reducing the 
number of beds available in the system, either 
by closing facilities or by funding fewer beds 
in existing facilities. However, the Ministry 
projects that the overall utilization rate will 
still be just 58% in 2012/13. 

• The average daily cost per youth varies signifi- 
cantly among custody/detention facilities. For 
example, in 2011, the average daily cost per 
youth ranged from $331 to $3,012 for agency- 
operated open facilities, from $475 to $1,642 
for agency-operated secure facilities, and 
from $1,001 to $1,483 for Ministry-operated 
secure facilities. 

• The Ministry's "single-case management" 
model has been a positive initiative. The aim 
is to have a youth's case assigned the same 
probation officer any time the youth is in the 
system. As well, in our review of case files, 
we noted many times where the knowledge 
and experience of probation officers was put 
to good use to manage youths' needs. How- 
ever, many of the required risk assessments 
and identified rehabilitation needs were not 
being documented. Also, many court-ordered 
conditions were not being complied with, or 
we could not determine compliance because 
there was not enough documentation or the 
conditions were unverifiable. 

• Ministry recidivism (reoffending) rates were 
35% for youths with community sentences 
and 59% for youths who had served custody 
sentences. However, these recidivism statistics 
exclude more than 80% of youths who have 
come into contact with the program. 
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Introduction 




fiscal results for the year ended March 31, 2012, 

including some details of what the government 

accomplished in the fiscal year. Providing such 

information enhances the fiscal accountability of 
Ontario's Public Accounts for each fiscal year end- 

the government to both the Legislative Assembly 

ing on March 31 are prepared under the direction 

and the public. 

of the Minister of Finance, as required by the Finan- 

The three supplementary volumes of the Public 

cial Administration Act (Act). The Public Accounts 

Accounts consist of the following: 

comprise the province's annual report, including 

• Volume 1 — statements from all ministries and 

the province's consolidated financial statements, 

a number of schedules providing details of the 

and three supplementary volumes of additional 

province's revenues and expenses, its debts 

financial information. 

and other liabilities, its loans and investments, 

The government's responsibility for preparing 

and other financial information; 

the consolidated financial statements involves 

• Volume 2 — audited financial statements of 

ensuring that the information, including the many 

significant provincial corporations, boards, 

amounts based on estimates and judgment, is pre- 

and commissions whose activities are 

sented fairly. The government is also responsible for 

included in the province's consolidated finan- 

ensuring that an effective system of control, with 

cial statements, as well as other miscellaneous 

supporting procedures, is in place to ensure that 

audited financial statements; and 

transactions are authorized, assets are safeguarded, 

, , , • Volume 3 — detailed schedules of ministry 

and proper records are maintained. 

payments to vendors and transfer-payment 

My Office audits these consolidated financial 

recipients. 

statements. The objective of our audit is to obtain 

My Office reviews the information in the prov- 

reasonable assurance that the statements are free of 

ince's annual report and in Volumes 1 and 2 of the 

material misstatement — that is, free of significant 

Public Accounts for consistency with the informa- 

errors or omissions. The consolidated financial 

tion presented in the province's consolidated finan- 

statements, along with my Independent Auditor's 

cial statements. 

Report, are included in the province's annual report. 

The Act requires that, except in extraordinary 

The province's 2011/12 annual report also con- 
circumstances, the government deliver its annual 

tains a Financial Statement Discussion and Analysis 

report to the Lieutenant Governor in Council within 

section that provides additional information 

180 days of the end of the fiscal year. The three 

regarding the province's financial condition and 
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supplementary volumes must be submitted to the 
Lieutenant Governor in Council within 240 days 
of the end of the fiscal year. Upon receiving these 
documents, the Lieutenant Governor in Council 
must lay them before the Legislative Assembly or, 
if the Assembly is not in session, make the informa- 
tion public and then lay it before the Assembly 
within 10 days of the time it resumes sitting. 

This year, the government released the prov- 
ince's 2011/12 Annual Report and Consolidated 
Financial Statements, along with the three Public 
Accounts supplementary volumes, on Septem- 
ber 13, 2012, meeting the 180-day deadline. 

In conducting our annual audit of the Public 
Accounts we work closely with the Ministry of 
Finance (Ministry) and particularly with the Office 
of the Provincial Controller. While we might not 
always agree on financial reporting issues, our 
working relationship has always been professional 
and constructive. 



The Province's 2011/1 
Consolidated Financial 
Statements 



The Auditor General Act requires that I report 
annually on the results of my examination of the 
province's consolidated financial statements. I am 
pleased to report that my Independent Auditor's 
Report to the Legislative Assembly on the province's 
consolidated financial statements for the year 
ended on March 31, 2012, is free of reservations. It 
reads as follows: 

Independent Auditor's Report 

To the Legislative Assembly of the Province of 
Ontario 

I have audited the accompanying consolidated 
financial statements of the Province of Ontario, 
which comprise the consolidated statement of 



financial position as at March 31, 2012, and the 
consolidated statements of operations, change 
in net debt, change in accumulated deficit, 
and cash flow for the year then ended and a 
summary of significant accounting policies and 
other explanatory information. 

Management's Responsibility for the Consoli- 
dated Financial Statements 
The Government of Ontario is responsible for 
the preparation and fair presentation of these 
consolidated financial statements in accord- 
ance with Canadian public sector accounting 
standards, and for such internal control as the 
Government determines is necessary to enable 
the preparation of consolidated financial state- 
ments that are free from material misstate- 
ment, whether due to fraud or error. 

Auditor's Responsibility 
My responsibility is to express an opinion on 
these consolidated financial statements based 
on my audit. I conducted my audit in accord- 
ance with Canadian generally accepted aud- 
iting standards. Those standards require that 
I comply with ethical requirements and plan 
and perform the audit to obtain reasonable 
assurance about whether the consolidated 
financial statements are free from material 
misstatement. 

An audit involves performing procedures to 
obtain audit evidence about the amounts and 
disclosures in the consolidated financial state- 
ments. The procedures selected depend on the 
auditor's judgment, including the assessment 
of the risks of material misstatement of the 
consolidated financial statements, whether 
due to fraud or error. In making those risk 
assessments, the auditor considers internal 
control relevant to the entity's preparation and 
fair presentation of the consolidated financial 
statements in order to design audit procedures 
that are appropriate in the circumstances, but 
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not for the purpose of expressing an opinion 
on the effectiveness of the entity's internal 
control. An audit also includes evaluating 
the appropriateness of accounting policies 
used and the reasonableness of accounting 
estimates made by the Government, as well 
as evaluating the overall presentation of the 
consolidated financial statements. 

I believe that the audit evidence I have 
obtained is sufficient and appropriate to pro- 
vide a basis for my opinion. 

Opinion 

In my opinion, these consolidated financial 
statements present fairly, in all material 
respects, the consolidated financial position 
of the Province of Ontario as at March 31, 
2012 and the consolidated results of its oper- 
ations, change in its net debt, change in its 
accumulated deficit, and its cash flows for the 
year then ended in accordance with Canadian 
public sector accounting standards. 

[signed] 

Jim McCarter, FCA 
Toronto, Ontario Auditor General 
September 6, 2012 Licensed Public Accountant 

The above audit opinion is without any reserva- 
tion, which indicates that the consolidated financial 
statements fairly present the province's fiscal results 
for the 2011/12 fiscal year and its financial position 
at March 31, 2012. This "clean" audit opinion 
means that based on our audit work, I can reason- 
ably conclude that the province's consolidated 
financial statements have been prepared in accord- 
ance with accounting standards the Canadian Insti- 
tute of Chartered Accountants (CICA) recommends 
for governments when preparing their financial 
statements. In other words, I am communicating 
to users that the province's consolidated financial 
statements do not have any material or significant 



errors and provide a fair reflection of what has 
actually transpired during the year. 

If I were to have significant concerns with the 
government's compliance with the CICA accounting 
standards, I would be required to issue an audit 
opinion with a reservation. An audit opinion with a 
reservation means significant financial transactions 
have either not been recorded or not been recorded 
properly in the province's consolidated financial 
statements. 

In determining whether a reservation is needed, 
auditors consider the materiality or significance of 
the unrecorded or misstated item in relation to the 
overall consolidated financial statements. An item 
is material if it is considered significant to financial 
statement users. An assessment of what is material 
(significant) and immaterial (insignificant) is based 
primarily on my professional judgment. Essentially, 
I ask the question "Is this error or misstatement sig- 
nificant enough that it could affect decisions made 
by users of the province's consolidated financial 
statements?" If the answer is yes, then I consider 
the error or omission material. 

To make this assessment I calculate a materiality 
threshold. Historically, this threshold has been set 
at around 0.5% of total government expenses or 
revenues for the year. If the misstated items indi- 
vidually or collectively exceed the threshold, and 
management was not willing to make the required 
adjustments, a reservation in my audit opinion 
would normally be required. However, no such 
reservation was required this year. 

As a final comment, I wish to point out that it 
is a notable achievement that in the past 19 years, 
all Ontario governments, regardless of the polit- 
ical party in power, have complied in all material 
respects with the CICA standards. Accordingly, my 
predecessor and I have been able to issue "clean" 
audit opinions on the province's consolidated 
financial statements since the province moved to 
adopt the CICA's Public Sector Accounting Board 
standards in the 1993/94 fiscal year. 
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Update on Ontario's Debt 
Burden 




In our 2011 Annual Report, we discussed the differ- 
ent measures of government debt — total debt, net 
debt, and accumulated deficit. We noted that the 
province's growing debt burden was attributable 
to government borrowing to finance recent large 
deficits and increased infrastructure spending. We 
compared Ontario's ratio of net debt-to-GDP to 
other Canadian and international jurisdictions, and 
highlighted the consequences to the province of 
carrying a large debt load. 

In updating our debt analysis this year, we 
observe that the province's debt, whether measured 
by total debt, net debt, or accumulated deficit con- 
tinues to increase, as illustrated in Figure 1. 

While the government has not provided details 
on its projected debt beyond the 2014/15 fiscal 
year, we estimate that if the government balances 
its books as it projects in 2017/18, Ontario's total 
debt, which represents the total amount of money 
the government owes to outsiders in the form of 
bonds issued in the public capital markets, non- 
public debt, T-bills and U.S. commercial paper, will 
still total more than $340 billion or double what the 
outstanding debt was at the end of 2007/08. Net 
debt, which is the difference between the govern- 
ment's total liabilities and its financial assets, will 
likely surpass $320 billion, and the accumulated 
deficit, which represents the sum of all past 



annual deficits and surpluses, will stand at around 
$210 billion by 2017/18. 

ONTARIO'S NET DEBT 

We noted last year that net debt is often considered 
the best measure of a government's fiscal situation. 
While it is important to examine whether net debt is 
increasing or decreasing over time, the level of debt 
relative to the size of the economy — that is, net 
debt to gross domestic product (GDP) — is generally 
considered to be a good indicator of a government's 
ability to manage its debt load. When this ratio is 
rising, it means the government's net debt is grow- 
ing at a faster rate than the provincial economy. 

Ontario's net debt-to-GDP ratio has risen stead- 
ily from a low of 26.8% in 2007/08 to 36.9% in 
the 2011/12 fiscal year, as illustrated in Figure 2. 
The projected growth in this ratio indicates that 
government debt will continue to grow at a faster 
rate than the provincial economy until 2015/16, 
when the expected growth rate of government 
debt will fall below the expected growth rate of the 
provincial economy. 

Another useful tool for assessing Ontario's 
debt load is to compare it with other Canadian 
jurisdictions. Figure 3 illustrates the net debt of the 
federal government and the provinces, along with 
their respective ratios of net debt to GDP, and the 
amount of debt owed by each resident of the juris- 
diction. As of March 31, 2012, with the exception 
of the province of Quebec and the federal govern- 
ment, Ontario has the highest net debt-to-GDP ratio 



Figure 1: Total Debt, Net Debt, and Accumulated Deficit, 2007/08-2014/15 ($ million) 

Source of data: 2012 Ontario Budget, 2011/12 Province of Ontario Annual Report and Consolidated Financial Statements, 2012 Ontario Economic Outlook 
and Fiscal Review, and the Office of the Auditor General of Ontario 



Actual Estimate 



2007/08" 2008/09" 2009/ 10" 2010/11" 2011/12" 2012/13° 2013/14 2014/15 

Total debt 162,217 176,915 212,122 236,629 257,278 278,000 297,500 a 314,100 a 

Net debt 156,616 169,585 193,589 214,511 235,582 257,600 277,600 c 293,300 c 

Accumulated deficit 105,617 113,238 130,957 144,573 158,410 172,800 185,600 c 195,700 c 



a. 2012 Ontario Budget 

b. 2011/12 Province of Ontario Consolidated Financial Statements 

c. 2012 Ontario Economic Outlook and Fiscal Review 



Public Accounts of the Province 



and the highest amount of debt owed by residents 
of a jurisdiction. 

CONSEQUENCES OF HIGH 
INDEBTEDNESS 

In our 2011 Annual Report, we highlighted that the 
province's escalating indebtedness has a number of 
negative consequences. These include: 

• debt-servicing costs taking away funding 
needed for other programs; 

• greater vulnerability to increases in interest 
rates; and 

• potential credit-rating downgrades and chan- 
ges in investor sentiment. 

At that time, some analysts saw little evidence 
to suggest that Ontario's credit rating needed to be 
downgraded, although some noted that Ontario's 
large borrowing requirements, along with its grow- 
ing reliance on foreign lenders, would increase 
this risk. However, there have been a number 
of developments in this regard since our 2011 
Annual Report. 

ONTARIO'S CREDIT RATING 

A credit rating is an assessment of a borrower's 
creditworthiness with respect to specified debt obli- 

Figure 2: Ontario Net Debt-to-GDP Ratio, 
2005/06-2017/18 (%) 

Source of data: 2012 Ontario Economic Outlook and Fiscal Review 
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gations. It indicates the capacity and willingness 
of a borrower to pay the interest and principal on 
these obligations in a timely manner. The province 
requires ratings from recognized credit-rating agen- 
cies to issue debt in capital markets. The three main 
credit-rating agencies are Moody's Investors Service 
(Moody's), Standard and Poor's (S&P), and DBRS. 

Credit-rating agencies assess a government's 
creditworthiness largely based on its capacity to 
manage its debts, and they consider such factors 
as that government's economic resources and 
prospects, institutional strengths, financial health, 
and susceptibility to major risks. Investors use this 
credit rating when making investment decisions. 

Credit ratings influence borrowing conditions by 
affecting both the cost and the availability of credit. 
A credit rating has an impact on the cost of future 
government borrowing because a lower rating indi- 
cates that the agency believes the risk of the govern- 
ment defaulting on its debt is higher, and investors 
will accordingly demand a greater risk premium in 
the form of a higher interest rate before they will 
lend to that jurisdiction. A rating downgrade can 
also result in a reduction of the potential market for 
a government's debt, as some investors are unable 
(due to contractual or institutional constraints) or 
unwilling to hold debt below a certain rating. 

Credit-rating agencies use letter designations 
to rate a jurisdiction's debt. For example, Moody's 
assigns credit ratings of Aaa, Aa, A, Baa, Ba, B, 
Caa, Ca, C, WR (withdrawn) and NR (not rated). 
Obligations rated Aaa are judged to be of the high- 
est quality and subject to the lowest level of credit 
risk, whereas obligations rated C are the lowest 
rated and are often in default, with little prospect 
for recovery of principal or interest. S&P and DBRS 
assign similar credit ratings ranging from AAA to D. 

In addition to a credit rating, the agency may 
issue a credit outlook that indicates the potential 
direction of a rating over the intermediate term, 
typically six months to two years. When determin- 
ing a rating outlook, the agency considers any 
changes in economic or fundamental business con- 
ditions. An outlook is not necessarily a precursor of 
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Figure 3: Net Debt ($), Net Debt to GDP (%) and Net Debt Per Capita ($) of Canadian Jurisdictions, 2011/12 and 
2010/11 

Source of data: 2011/12 Province of Ontario Annual Report and Consolidated Financial Statements, 2012 Federal Budget and budget updates, 
2011/12 Annual Report and Consolidated Financial Statements of other provinces, 2012 provincial budgets, Statistics Canada, and the 
Office of the Auditor General of Ontario 
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a rating change but rather informs investors about 
the agency's view of the potential evolution of a rat- 
ing — either up or down. A positive outlook means 
that a rating might be raised. A negative outlook 
means that a rating might be lowered, and a stable 
outlook means that a rating is not likely to change 
in the short term. 

All three credit-rating agencies updated their 
assessment of the province's credit rating shortly 
after the government released its 2012 budget. 
Moody's had put the province on a credit watch in 
December 2011, and in April 2012 it downgraded 
Ontario's credit rating from Aal to Aa2. On the 
other hand, in April 2012 S&P revised its outlook 
on Ontario to "negative" from "stable" and main- 
tained the province's current AA- rating, which it 
had downgraded in 2009. DBRS has not changed 
Ontario's rating since downgrading it to AA (low) 
in the fall of 2009. It may well be that Moody's 
downgrade is a catch-up to the S&P and DBRS 
downgrades in late 2009. Ontario's rating relative 
to other Canadian senior governments is shown in 
Figure 4. 



IMPACT OF LOWER CREDIT RATING/ 
REVISED OUTLOOK 

While downgrades and poorer outlooks for the 
provinces' credit ratings theoretically increase a 
government's future borrowing costs, there is no 
evidence yet suggesting these latest ratings have 
had a significant impact on Ontario's borrowing 
costs. Ontario's bond interest costs have remained 
relatively unchanged since the ratings were 
revised, which indicates investors still want to hold 
Ontario debt. 

Ontario bonds remain relatively attractive 
because many other jurisdictions around the world 
have been affected by the 2008 global financial 
downturn to a greater extent, and investors are 
therefore reluctant to invest in these jurisdictions. 
The credit-rating agencies also believe that Ontario 
has the necessary fiscal flexibility to improve its 
financial position over the medium term, so have 
indicated they expect no more rating adjustments 
in the near future. 
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Figure 4: Credit Rating by Province and Balanced Budget Target Date 

Source of data: Provincial and federal budget documents, and Laurentian Provincial Monitor, as of April 26, 2012 
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The government of Canada is also one of the 
few remaining jurisdictions in the world that has 
retained its Aaa/ AAA credit rating — the highest 
that can be assigned. This means demand for Can- 
adian government debt, both federal and provin- 
cial, is high, especially among investors looking for 
relatively risk-free investments. This demand works 
to push down interest rates. And, because investors 
associate Ontario debt with the perceived credit- 
worthiness of the federal government, Ontario 
benefits from the relative strength of investor faith 
in federal government debt. 

Even with its most recent downgrade, Ontario 
has the same credit rating as Quebec and the 
Maritime provinces, with a Moody's credit rating of 
Aa2. Only the federal government and the Western 
provinces have higher ratings. 

CONCLUSION 

In our 2011 Annual Report we concluded that while 
the government had presented a plan to eliminate 
its annual deficits by the 2017/18 fiscal year, no 
strategy had been presented for paying down its 
existing and future debt. We indicated at the time 
that once deficits have been tackled, one strategy for 
paying down debt would be to hold the line on any 



future debt increases and use additional revenues 
generated by a growing economy to start reducing 
the debt. We went on to say that the government 
should consider providing legislators and the public 
with long-term targets and a strategy for how to 
address the current and projected debt burden. 
These comments continue to be appropriate. 



In order to meet the Province's fiscal targets, the 
government is managing spending. Last year, 
growth in program spending was less than 1% — 
the second-lowest rate of growth in Ontario in a 
decade. For the fourth year in a row, Ontario is 
ahead of its targets in lowering the deficit. The 
Province's deficit for the 2012/13 fiscal year is 
projected to be $14.4 billion, an improvement of 
$0.4 billion from the 2012 Budget forecast. 

The Province's debt-to-GDP ratios are 
expected to increase due to the projected 
deficits and investments in capital. The ratios 
will stabilize and begin to decline as the deficit 
is eliminated. The net debt-to-GDP ratio is 
expected to peak in 2014/15 at 41.2%, which is 
below the 41.3% forecast in the 2012 Budget. 
The accumulated deficit-to-GDP ratio is 
expected to peak at 27.5% in 2014/15. 
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Update on the Unfunded 
Liability of the Workplace 
Safety and Insurance Board 



The Workplace Safety and Insurance Board (WSIB) 
is a statutory corporation created by the Workplace 
Safety and Insurance Act, 1997 (Act). Its primary 
purpose is to provide income support and medical 
assistance to workers injured on the job. The WSIB 
receives no funding from government; it is financed 
through premiums on employer payrolls. 

Over the past decade, our annual reports have 
made a number of references to our concerns about 
the significant growth in the WSIB's unfunded 
liability, which is the difference between the value 
of the WSIB's assets and its estimated financial 
obligations to pay benefits to injured workers. Our 
2009 Annual Report included a separate section 
that discussed our review of the WSIB's unfunded 
liability. In that section we expressed our concern 
that the growth and magnitude of the unfunded 
liability posed a risk to the system's financial viabil- 
ity and ultimately could result in the WSIB being 
unable to meet its existing and future commitments 
to provide worker benefits. 

We also urged the government to reconsider 
the exclusion of the WSIB's financial results from 
the province's financial statements. Excluding 
these financial results is based on the WSIB's clas- 
sification as a "trust"; however, given its significant 
unfunded liability and various other factors, we 
questioned whether the WSIB was operating like a 
true trust. Including the WSIB in the government's 
financial reporting would have a significant impact 
on the government's fiscal performance. 

Despite efforts taken by management to improve 
the operations of the insurance fund, the WSIB's 
unfunded liability has continued to grow, princi- 
pally due to a significant change to the discount 
rate introduced in 2011 that added $2 billion to the 
liability. Absent the impact of this change, the WSIB 
would have showed its first surplus in 10 years. 



As of December 31, 2011, the unfunded liability 
totalled $14.2 billion, an increase of $2 billion from 
its December 31, 2010, balance of $12.4 billion. The 
WSIB's funding ratio — the percentage of assets to 
liabilities— was 52.2% as of December 31, 2011. (As 
of December 31, 2010, it had been 54.7%.) 

In September 2010, the WSIB announced an 
independent funding review to provide advice on 
how to best ensure the long-term financial viabil- 
ity of Ontario's workplace safety and insurance 
system. The May 2012 report by Professor Harry 
Arthurs was comprehensive and contained a series 
of recommendations to ensure the WSIB remains 
financially stable and sufficiently funded. 

In particular, the report recommended a new 
funding strategy for the WSIB, including the follow- 
ing key elements: 

• realistic assumptions, including a discount 
rate based on the best actuarial advice; 

• moving the WSIB as quickly as feasible beyond 
a "tipping point" of a 60% funding ratio (tip- 
ping point being defined as a crisis in which 
the WSIB could not within a reasonable time 
frame and by reasonable measures generate 
sufficient funds to pay workers' benefits) ; and 

• putting the WSIB on course to achieve a 
90%-110% funding ratio within 20 years. 

In response to our concerns and to the recom- 
mendations of the Arthurs report, in June 2012 the 
government filed Regulation 141/12 under the Act, 
which, effective January 1, 2013, will require the 
WSIB to ensure it meets the following funding suf- 
ficiency ratios by specified dates: 

• 60% on or before December 31, 2017 

• 80% on or before December 31, 2022 

• 100% on or before December 31, 2027 

The regulation also requires the WSIB to submit 
a plan to the Minister of Labour by June 30, 2013, 
outlining the measures it will take to achieve 
these goals by the prescribed dates. As reported 
in our 2011 Annual Report follow-up section, the 
WSIB continues to take actions that have recently 
improved its operating results and financial pos- 
ition. For example, during 2011, the WSIB reported 
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its first operational surplus in 10 years (excluding 
the change in the discount rate discussed below) . 
Increased premium revenue, fewer new lost-time 
claims, more workers returning to work sooner, 
and fewer workers suffering permanent impair- 
ment have not only contributed to these financial 
improvements but have demonstrated the WSIB's 
focus on getting injured workers back to work as 
quickly as possible. 

However, the WSIB's ability to achieve the fund- 
ing sufficiency ratios prescribed in the regulation 
will be affected by changes to certain actuarial 
assumptions and proposed new accounting 
standards. 

• The discount rate is the interest rate used to 
calculate future benefit obligations in cur- 
rent dollars. In layman's terms, it is often 
perceived as the expected rate of return an 
insurance company or pension fund can earn 
on its investments. During 2011 a discount 
rate reduction from 7.0% to 5.5% resulted in 
an almost $2 billion increase in the unfunded 
liability. WSIB management have concluded 
that subject to any changes in accounting 
standards (as discussed below), it would be 
appropriate to retain this 5.5% discount rate 
for the period 2012-2015 and increase it to 
6.0% thereafter. For example, reducing the 
discount rate by one percentage point would 
add approximately $2.1 billion to the benefits 
obligation. 

• The WSIB reports its financial results based on 
International Financial Reporting Standards 
(IFRS) . A proposed new standard for valuing 
insurance liabilities is expected to take effect 
as of the end of fiscal 2016. If approved in its 
current form, it could increase the unfunded 
liability significantly because it would require 
a further reduction in the discount rate WSIB 
uses to calculate its benefits obligation. Based 
on current market interest rates, this could 
increase the actuarial estimate of WSIB's 
benefit obligations by an estimated $3 billion. 
The new standard also proposes that a new 



risk margin be included in the benefits obliga- 
tion that could increase the benefits obligation 
by a further $1 billion. (This new risk margin 
is similar to the actuarial provision for adverse 
deviations, which is a reserve for the likely 
difference between the actual result of a 
calculation and the corresponding result using 
best estimate assumptions.) Without any com- 
pensating increases in the WSIB's premiums 
rate, its investment performance or further 
success in reducing lost-time claims to reduce 
costs, the combined $4 billion impact of this 
proposed standard could significantly impair 
the WSIB's ability to meet its 2017 legislated 
funding ratio of 60%. 
As a result of the government's commitments to 
address its unfunded liability, we support the con- 
tinued classification of the WSIB as a trust for the 
2011/12 fiscal year and therefore the exclusion of 
the unfunded liability from the province's liabilities. 
However, we will continue to monitor the progress 
being made toward meeting the funding sufficiency 
ratios prescribed by the regulation. Should we feel 
enough progress is not being achieved, we will re- 
evaluate our position. 

| WSIB RESPONSE 

The proposed new IFRS standard for valuing 
insurance liabilities will have the effect of valu- 
ing liabilities using the market rates prevailing 
at discrete points in time. This introduces 
volatility as well as one-time changes of a very 
large magnitude, both positive and negative, 
depending on whether interest rates are rising 
or falling. 

WSIB is following IFRS standards in prepar- 
ing its financial statements. However, for pur- 
poses of complying with the funding sufficiency 
ratios prescribed by regulation, we believe that 
IFRS standards are not well suited to funding 
a going concern insurance fund such as the 
WSIB's. Consequently, the WSIB will consult 
with the government on the merits of adopting 
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a "going-concern basis" to value the benefits 
liabilities of the insurance fund, for funding pur- 
poses. This will have the effect of using a long- 
term expected rate-of-investment return with 
which to value liabilities. This method will avoid 
potentially large multibillion-dollar swings in 
the valuation of liabilities from year to year and 
the consequent instability for employers who 
must fund the system and the workers who must 
rely on it. 



Update on the Electricity 
Sector Stranded Debt 



In Section 3.04 of our 2011 Annual Report, we 
commented on the stranded debt of the electricity 
sector and the Debt Retirement Charge (DRC), a 
component of nearly every Ontario ratepayer's 
electricity bill. 

The stranded debt arose with the passage of 
the Energy Competition Act, 1998, which resulted 
in a major restructuring of the electricity industry, 
including the breakup of the old Ontario Hydro 
into three main successor companies: Hydro One, 
Ontario Power Generation (OPG) and the Ontario 
Electricity Financial Corporation (OEFC) . OEFC 
was given the responsibility to manage the legacy 
debt of the old Ontario Hydro and certain other 
liabilities not transferred to Hydro One and OPG. 

OEFC inherited $38.1 billion in total debt and 
other liabilities from Ontario Hydro when the elec- 
tricity market was restructured on April 1, 1999. 
Only a portion of the $38.1 billion was supported 
by the value of the assets of Hydro One, OPG and 
the Independent Electricity System Operator, leav- 
ing $20.9 billion of stranded debt not supported 
by assets. 

The government's long-term plan to service and 
retire the $20.9 billion in stranded debt involved 
dividing it into two components: 



• An estimated $13.1 billion was to be sup- 
ported through future revenue streams 
from payments in lieu of taxes made by the 
electricity-sector companies (OPG, Hydro One 
and the municipal electrical utilities), and 
from the cumulative annual combined profits 
of OPG and Hydro One in excess of the gov- 
ernment's $520-million annual interest cost of 
its investment in the two companies. 

• The remaining $7.8 billion, called the resid- 
ual stranded debt, was the estimated portion 
of the stranded debt that could not be sup- 
ported by the expected dedicated revenue 
streams from the electricity companies. The 
Electricity Act, 1998 authorized a new Debt 
Retirement Charge (DRC), to be paid by elec- 
tricity ratepayers, until the residual stranded 
debt was retired. 

This structure was intended to eliminate the 
stranded debt in a prudent manner while distribut- 
ing the debt repayment burden between electricity 
consumers and the electricity sector. 

Collection of the DRC began on May 1, 2002. 
The rate was established at 0.7 cents per kilo- 
watt hour (kWh) of electricity and remains the 
same today. Currently, the OEFC collects more 
than $940 million a year in DRC revenue. As of 
March 31, 2012, approximately $9.7 billion in DRC 
revenue had been collected. 

Our 2011 Annual Report focused on providing 
details about how much DRC revenue has been 
collected, the progress in eliminating the residual 
stranded debt, and when electricity ratepayers 
might expect to see the DRC eliminated. 

Section 85 of the Electricity Act, 1998 (Act) 
entitled "The Residual Stranded Debt and the 
Debt Retirement Charge" gave the government the 
authority to implement the DRC, and this same 
section specifies when it is to end. The key observa- 
tions from our 2011 Annual Report were based on 
our interpretations of the provisions of section 85 
of the Act and assessing whether these provisions 
had been complied with in both spirit and form. 
Specifically, section 85 requires that the Minister 
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of Finance determine the residual stranded debt 
"from time to time" and to make these determina- 
tions public. When the Minister determines that the 
residual stranded debt has been retired, collection 
of the DRC must cease. 

While the Act did not specify precisely how 
the determination of the residual stranded debt 
was to be done, it does allow the government, by 
regulation, to establish what is to be included in its 
calculation. We also observed that the term "from 
time to time" was not formally defined, and could 
be left solely up to the government of the day to 
determine. Since the passage of the Act more than 
a decade ago, we noted the Minister had made 
no such public determination of the outstanding 
amount of the residual stranded debt. Our view 
was that the intent of section 85 was that Ministers 
had an obligation to provide a periodic update to 
ratepayers on what progress their payments were 
having on paying down the residual stranded debt. 
We concluded that a decade was long enough, and 
suggested the Minister should provide ratepayers 
with an update. 

In response to these observations, the govern- 
ment introduced Regulation 89/12 under the Act 
on May 15, 2012, to provide transparency and meet 
reporting requirements on the outstanding amount 
of residual stranded debt. The new regulation 
formally establishes how the residual stranded debt 
is to be calculated, and requires annual reporting of 
the amount in The Ontario Gazette. 

We were pleased to see this increased level of 
transparency was also reflected in the 2012 budget, 
which contained a residual stranded debt estimate 
of $5.8 billion as of March 31, 2011, and a projected 
estimate of $4.5 billion as of March 31, 2012. 
Prior to the 2012 budget, there had been no public 
update provided on the estimated residual stranded 
debt since the $7.8 billion estimate on April 1, 
1999. The update in the 2012 Ontario Economic 
Outlook and Fiscal Review also contained the chart, 
illustrated in Figure 5, reflecting annual residual 
stranded debt estimates back to April 1, 1999, and 
amounts going up to March 31, 2012. 



Figure 5: Residual Stranded Debt and 0EFC Unfunded 
Liability for Each Fiscal Year Since 1999 ($ billion) 

Source of data: 2012 Ontario Economic Outlook and Fiscal Review 
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Use of Legislation to Override 
Accounting Standards 



One significant recent development in Canadian 
public accounting is that some governments have 
legislated specific accounting treatments that 
would apply in certain circumstances rather than 
relying on established generally accepted account- 
ing principles to prescribe how transactions would 
be recorded. On several occasions in recent years 
the Ontario government has done this bypassing 
legislation or regulations that give it the authority 
to prescribe specific accounting policies for its pub- 
lic sector entities. 

We first raised concerns about this practice in 
our 2008 Annual Report, where we warned that it 
was a troubling precedent to establish accounting 
principles through legislation rather than through 
an independent, consultative process such as that 
followed by the Public Sector Accounting Board 
(PSAB) of the Canadian Institute of Chartered 
Accountants (CICA) . Although this and subsequent 
developments have not yet resulted in the prov- 
ince's consolidated financial statements containing 
a material departure from PSAB standards, the risk 
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of such departures in future financial statements 
has increased. Here is a chronological synopsis of 
these developments since 2008: 

• The Investing in Ontario Act, 2008 (Act) and 
related regulations allowed the government 
to reduce its accumulated deficit and provide 
additional transfers to eligible recipients from 
unplanned surpluses reported in its consoli- 
dated financial statements. Any transfers 
made under the Act would be recorded as an 
expense of the government for that fiscal year 
irrespective of PSAB accounting standards. 

• In the 2009/10 fiscal year, the Education Act 
and the Financial Administration Act were 
amended. Amendments under the Education 
Act specified that the government can pre- 
scribe the accounting standards that school 
boards use in preparing financial statements. 
Amendments under the Financial Administra- 
tion Act allow the government to specify the 
accounting standards to be used by any public 
or non-public entity whose financial state- 
ments are included in the province's consoli- 
dated financial statements. School boards and 
certain other government organizations are 
affected by both these legislative amendments. 

• In 2011, a regulation under the Financial 
Administration Act directed Hydro One to 
prepare its financial statements in accordance 
with U.S. generally accepted accounting prin- 
ciples, effective January I, 2012. The govern- 
ment has since provided the same direction 
to OPG. American accounting rules allow for 
rate-regulated entities to defer expenses to 
future years; the government's direction to 
follow U.S. rules came in anticipation of the 
planned Canadian adoption of International 
Financial Reporting Standards, which do 
not provide for the use of rate-regulated 
accounting. 

• Ontario government regulations now require 
capital transfers and transfers of tangible 
capital assets to be accounted for by transfer 
recipients as deferred capital contributions. 



The amounts transferred are to be brought 
into revenue at the same rate as amortization 
expense is recognized on the related assets. 
We have historically supported this account- 
ing as we believe that in most instances it 
complies with generally accepted accounting 
principles. However, PSAB standards are open 
to interpretation in this respect, so to ensure 
consistency the government considered it 
necessary to introduce a regulation requiring 
this treatment. 
• The direction taken by the province beginning 
in 2008 to legislate accounting standards 
was further supported in the Strong Action 
for Ontario Act (Budget Measures), 2012, the 
latest amendments to the Financial Admin- 
istration Act. These amendments provide 
the government with full authority to make 
regulations regarding the accounting policies 
and practices used to prepare its consolidated 
financial statements. 
We believe it is critical that Ontario continue to 
prepare its financial statements in accordance with 
generally accepted PSAB standards. 

As the auditor of the province's consolidated 
financial statements, I am required by the Auditor 
General Act to provide an opinion on "whether 
the consolidated financial statements of Ontario, 
as reported in the Public Accounts, present fairly 
information in accordance with appropriate 
generally accepted accounting principles." If I 
conclude that the government's reported deficit or 
surplus under legislated accounting standards is 
significantly different than what it would be under 
generally accepted accounting standards, I will 
need to include a reservation in my audit opinion. 
However, based on the past 19 consecutive years of 
"clean" audit opinions, I am hopeful that this situa- 
tion will not arise. 
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Future Public Accounts 
Issues 



■ 

Most Canadian governments use PSAB standards in 
preparing their annual budgets, printed estimates, 
economic updates and year-end consolidated 
financial statements. When governments use the 
same set of accounting standards to prepare their 
key financial reports, the public can evaluate the 
government's expected financial performance 
against actual results and against the results of 
other jurisdictions. PSAB standards are intended to 
help governments publicly demonstrate steward- 
ship over the resources they manage, and thereby 
strengthen accountability to taxpayers. 

ACCOUNTING STANDARDS FOR 
GOVERNMENTS 

Accounting standards specify how and when 
transactions and other events are to be recognized, 
measured and disclosed in financial statements. 
To be objective and credible, accounting standards 
should be established by an independent, recog- 
nized professional body using a comprehensive, 
open and transparent standard-setting process. 

The Public Sector Accounting Board of the Can- 
adian Institute of Chartered Accountants is respon- 
sible for establishing accounting standards for the 
public sector. PSAB standards represent generally 
accepted accounting principles for governments in 
Canada and are the primary source of guidance for 
public-sector accounting. 

The PSAB emphasizes due process to ensure that 
the views of all interested parties are heard and 
considered, thereby maintaining the objectivity of 
the standard-setting process. In developing or revis- 
ing an accounting standard, the PSAB generally 
follows these five steps: 

• basic research; 

• approval of a project proposal; 



• issuing a statement of principles to a desig- 
nated group of accountants and non-account- 
ants for feedback; 

• issuing one or more public exposure drafts 
and soliciting comments from all interested 
individuals and organizations; and 

• approving and publishing a final standard. 

Canada is generally regarded as a public sec- 
tor world leader with respect to the consistent 
application of generally accepted accounting 
standards at all three levels of government. How- 
ever, maintaining this leadership role will not be 
without challenges. For instance, in the past few 
years, Canadian governments, including Ontario, 
have become concerned with several of the PSAB's 
accounting and financial reporting proposals. In 
the next section, we discuss two areas — financial 
instruments and rate-regulated accounting — where 
they have questioned whether proposed standards 
adequately reflect the unique environment in which 
governments operate when making decisions on 
financial reporting, budgeting and fiscal policy. 

Financial Instruments 

The PSAB's project to develop a new standard for 
the reporting of financial instruments began in 
January 2005 with the creation of a PSAB task 
force. Examples of financial instruments include 
debt, and derivatives such as currency swaps and 
foreign-exchange forward contracts. One of the key 
issues the task force had to address was whether 
changes in the fair value of derivative contracts 
held by a government should be reflected in its 
financial statements and, in particular, whether 
such changes should affect the government's 
annual surplus or deficit. 

In March 2011, the PSAB approved a new 
standard effective for fiscal periods beginning 
on or after April 1, 2015. The new public-sector 
accounting standard — PS 3450 "Financial Instru- 
ments" — provides guidance on the recognition, 
measurement, presentation and disclosure of gov- 
ernment financial instruments, and is similar to the 
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standards applicable to the private sector. One of its 
main requirements is that certain financial instru- 
ments, including derivatives, are to be recorded 
at fair value, and any unrealized gains or losses 
must be recorded annually in a new statement of 
remeasurement gains and losses. 

Ontario, along with certain other Canadian 
governments, did not endorse the introduction of 
fair-value remeasurements and the recognition of 
unrealized gains and losses on its derivative hold- 
ings. Ontario's view is that derivatives are used 
solely to mitigate foreign currency and interest-rate 
risks related to its long-term-debt holdings and 
that it has both the intention and ability to hold 
its derivatives until the associated debt matures. 
Accordingly, remeasurement gains and losses 
would net out to zero over the period to matur- 
ity and therefore have no long-term economic 
impact on annual government resource inflows or 
outflows. The government argues that recording 
paper gains and losses each year would reintroduce 
the very volatility the derivatives were acquired to 
avoid in the first place. 

Therefore, its view is that the inclusion of fair- 
value gains and losses in a government's financial 
reports, even if reported outside the statement of 
operations, does not reflect the economic substance 
of government financing transactions and does not 
meet the public's needs for transparent information 
on government finances. 

It should be noted that the PSAB is commit- 
ted to reviewing these standards on or before 
December 31, 2013, and has noted that its Concepts 
Underlying Financial Performance project might 
identify issues that need to be addressed within the 
financial instrument standard. 



| MINISTRY RESPONSE 




Ontario supports public-sector accounting stan- 
dards that promote transparency and account- 
ability in reporting on how taxpayer dollars 
are spent, and that support sound fiscal-policy 
decisions. Concerns with the introduction of 



fair-value accounting concepts into government 
financial reports are based on the fact that they 
will cause significant fluctuations in the recorded 
value of derivatives on a year-over-year basis as 
a result of movements in interest and exchange 
rates. Such results would, in turn, introduce 
difficult-to-explain fluctuations in the province's 
net debt, a key measure upon which both invest- 
ors and credit-rating agencies rely when assess- 
ing the financial health of the province. 



Rate-regulated Accounting 

Over the past four years, we have raised concerns 
about the appropriateness of recognizing rate- 
regulated assets and liabilities in the government's 
consolidated financial statements. Rate-regulated 
accounting practices were developed to recognize 
the unique nature of regulated entities such as elec- 
tricity generators, transmitters, and distributors. 
Under rate-regulated accounting, a regulator estab- 
lished under legislation, such as the Ontario Energy 
Board, approves the prices that a regulated entity 
may charge customers, and often allows regulated 
entities to defer certain costs for recovery in future 
periods. Such costs are typically set up as assets 
that, under normal generally accepted accounting 
principles, would be expensed in the year incurred. 

Ontario's electricity sector has three major prov- 
incially owned organizations — OPG, Hydro One 
and the OPA — that use rate-regulated accounting 
and whose financial position and operating results 
are included in the government's consolidated 
financial statements. The use of rate-regulated 
accounting by certain rate-regulated entities, while 
still allowed under Canada's generally accepted 
accounting principles, could be disallowed in the 
near future. PSAB standards currently allow OPG 
and Hydro One, which are defined as government 
business enterprises, to be included in the prov- 
ince's consolidated financial statements without 
their accounting policies being adjusted to remove 
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the impact of rate-regulated accounting. Given the 
PSAB's position, we accepted this accounting treat- 
ment even though we questioned whether rate- 
regulated assets and liabilities met the definition of 
bona fide assets or liabilities for the purposes of the 
government's consolidated financial statements. 

However, the OPA does not meet the PSAB 
criteria of being a government business enterprise, 
so the impact of rate-regulated accounting on its 
results should have been removed before the OPA 
was included in the consolidated statements. (This 
did not have a material impact on the province's 
reported results and therefore did not affect our 
audit opinion). 

Last year we reported that the era of rate- 
regulated accounting appeared to be ending for 
jurisdictions such as Canada that were converting 
to International Financial Reporting Standards. 
In January 2012, Canada's Accounting Standards 
Board reaffirmed that all government business 
enterprises should prepare their financial state- 
ments in accordance with IFRS for fiscal years 
beginning on or after January 1, 2012, which would 
mean that accounting for rate-regulated balances 
would no longer be allowed under Canadian 
accounting standards. 

However, since that time, the rate-regulated 
landscape has changed again. The accounting 
standard-setter in the United States, the Financial 
Accounting Standards Board, has not adopted IFRS 
and therefore continues to allow rate-regulated 
accounting. Partly in an effort to reconcile U.S. 
generally accepted accounting principles with IFRS, 
in March 2012 Canada's Accounting Standards 
Board granted a one-year extension, to January 1, 
2013, to the mandatory IFRS changeover date for 
entities with qualifying rate-regulated activities. In 
September 2012, it granted an additional one-year 
extension, to January 1, 2014. Therefore, PSAB 
accounting standards will continue to allow OPG 
and Hydro One to use rate-regulated accounting for 
inclusion in the province's consolidated financial 
statements until that time. Most recently, in late 
September 2012, the International Accounting 



Standards Board (IASB) decided to restart its pro- 
ject on rate-regulated accounting to consider the 
use of this accounting method under IFRS. 

Ontario's Ministry of Finance contends that the 
province's rate-regulated assets and liabilities might 
meet PSAB standards without reference to any of the 
rate-regulated provisions from Canada's Accounting 
Standards Board. As the Ministry is aware, we do 
not agree with this position. In its March 31, 2012, 
Annual Report and Consolidated Financial State- 
ments, the government commented specifically on 
rate-regulated accounting, noting that: 

Under these practices, a regulated entity 
may defer certain costs that are expected 
to be recovered in future, creating an asset 
on its balance sheet. Without rate-regu- 
lated accounting, these costs would be 
expensed in the year they were incurred, 
which could result in significant fluctua- 
tions in consumer electricity rates. 

The government recently passed a regulation 
allowing for and subsequently directing both Hydro 
One and OPG to prepare their future financial 
statements in accordance with U.S. generally 
accepted accounting principles, which allow for 
rate-regulated accounting. In the government's 
view, these regulations are required because the 
IASB has not yet completed its deliberations on the 
future of rate-regulated accounting. We are not the 
auditors of Hydro One or OPG, and so our concern 
is not the impact of this regulation on their individ- 
ual financial statements. However, with the deferral 
of the issue to January 1, 2014, it is the effect their 
accounting policies might have on the province's 
consolidated financial statement of March 31, 2014, 
and subsequent statements, that is our concern. 



| MINISTRY RESPONSE 




The Ministry recognizes the ongoing challenges 
in the accounting-standards-setting environ- 
ment to achieve consensus on the required 
approach for rate-regulated accounting. Given 



2012 Annual Report of the Office of the Auditor General of Ontario 



the deferral by the standards-setters to resolve 
this issue, the government had directed Hydro 
One and OPG to follow U.S. GAAP to allow the 
entities to continue to account for their rate- 
regulated assets and liabilities consistent with 
historical Canadian GAAP. This decision was 
consistent with actions by both the Canadian 
Securities Administrators and the Ontario 
Securities Commission that have enabled 
rate-regulated utilities to submit their financial 
statements on a U.S. GAAP basis until 2014. This 
also helped to ensure consistency in reporting 
among the province's rate-regulated entities 
until outstanding issues are resolved. 

In late September 2012, the International 
Accounting Standards Board decided to restart 
its project on rate-regulated accounting. The 
government looks forward to the standards-set- 
ters undertaking this review and, ultimately, to 
resolution of the outstanding certainty regarding 
the future of rate-regulated accounting. 



Statutory Matters 




Under section 12 of the Auditor General Act, I am 
required to report on any Special Warrants and 
Treasury Board Orders issued during the year. In 
addition, section 91 of the Legislative Assembly Act 
requires that I report on any transfers of money 
between items within the same vote in the Esti- 
mates of the Office of the Assembly. 

LEGISLATIVE APPROVAL OF 
EXPENDITURES 

Shortly after presenting its budget, the government 
tables detailed Expenditure Estimates in the Legisla- 
tive Assembly outlining, on a program-by-program 
basis, each ministry's spending proposals. The 
Standing Committee on Estimates (Committee) 



reviews selected ministry estimates and presents a 
report on them to the Legislature. The estimates of 
those ministries that are not selected for review are 
deemed to be passed by the Committee and are so 
reported to the Legislature. Orders for Concurrence 
for each of the estimates reported on by the Com- 
mittee are debated in the Legislature for a maximum 
of two hours and then voted on. For a more in-depth 
discussion of the legislative estimates review pro- 
cess, see section 3.07 of this Annual Report. 

Once the Orders for Concurrence are approved, 
the Legislature provides the government with legal 
spending authority by approving a Supply Act, 
which stipulates the amounts that can be spent by 
ministry programs, typically those set out in the 
estimates. Once the Supply Act is approved, the 
individual program expenditures are considered 
to be Voted Appropriations. The Supply Act, 2012, 
which pertained to the fiscal year ended March 31, 
2012, received Royal Assent on April 24, 2012. 

The Supply Act does not typically receive Royal 
Assent until after the start of the fiscal year — and 
sometimes even after the related fiscal year — but 
ministry programs require interim spending 
authority prior to its passage. For the 2011/2012 
fiscal year, the Legislature authorized these pay- 
ments by passing two acts allowing interim appro- 
priations: the Interim Appropriation for 2011-2012 
Act, 2010; and the Supplementary Interim Appro- 
priation Act, 2011 . These two acts received Royal 
Assent on December 8, 2010, and May 12, 2011, 
respectively, and authorized the government to 
incur up to $114.5 billion in public service expendi- 
tures, $3.5 billion in investments, and $193.5 mil- 
lion in legislative offices expenditures. Both acts 
were made effective as of April 1, 2011. 

The two interim acts provided the government 
with sufficient temporary appropriations to allow 
it to incur expenditures from April 1, 2011, to 
when the Supply Act, 2012 received Royal Assent 
on April 24, 2012. As the legal spending authority 
under the interim acts was intended to be tempor- 
ary, they were repealed under the Supply Act, 2012, 
and the authority to incur expenditures provided 
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under them was subsumed into the authority pro- 
vided under the Supply Act, 2012, which increased 
total authorized legislative offices expenditures 
from $193.5 million to $197 million. 

SPECIAL WARRANTS 

If the Legislature is not in session, section 1.0.7 of 
the Financial Administration Act allows for the issu- 
ance of Special Warrants authorizing the incurring 
of expenditures for which there is no appropriation 
by the Legislature or for which the appropriation 
is insufficient. Special Warrants are authorized 
by Orders-in-Council and approved by the Lieu- 
tenant Governor on the recommendation of the 
government. 

No Special Warrants were issued for the fiscal 
year ended March 31, 2012. 

TREASURY BOARD ORDERS 

Section 1.0.8 oi the Financial Administration Act 
allows the Treasury Board to make an order author- 
izing expenditures to supplement the amount of 
any voted appropriation that is expected to be 
insufficient to carry out the purpose for which 
it was made. The order may be made only if the 
amount of the increase is offset by a corresponding 
reduction of expenditures to be incurred from other 
voted appropriations not fully spent in the fiscal 
year. The order may be made at any time before 
the books of the government for the fiscal year are 
closed. The government considers the books to be 
closed when any final adjustments arising from our 
audit have been made and the Public Accounts have 
been tabled in the Legislature. 

Even though the Treasury Board Act, 1991 
was repealed and re-enacted within the Financial 
Administration Act in December 2009, subsection 
5 (4) of the repealed act was retained and allows 
the Treasury Board to delegate to any member 
of the Executive Council or to any public servant 
employed under the Public Service of Ontario Act, 
2006 any power, duty, or function of the Treasury 



Board, subject to limitations and requirements that 
the Treasury Board may specify. This delegation 
will continue to be in effect until replaced by a new 
delegation. For the fiscal year ended March 31, 
2012, the Treasury Board delegated its authority 
to ministers for issuing Treasury Board Orders to 
make transfers between programs within their min- 
istries, and to the Chair of the Treasury Board for 
making transfers in programs between ministries 
and making supplementary appropriations from 
contingency funds. Supplementary appropriations 
are Treasury Board Orders in which the amount 
of an appropriation is offset by a reduction to the 
amount available under the government's centrally 
controlled contingency fund. 

Figure 6 summarizes the total value of Treasury 
Board Orders issued for the past five fiscal years. 

Figure 7 summarizes Treasury Board Orders for 
the fiscal year ended March 31, 2012, by month 
of issue. 

According to the Standing Orders of the Legis- 
lative Assembly, Treasury Board Orders are to 
be printed in The Ontario Gazette, together with 
explanatory information. Orders issued for the 
2011/12 fiscal year are expected to be published in 
The Ontario Gazette in the near future. A detailed 
listing of 2011/12 Treasury Board Orders, showing 
the amounts authorized and expended, is included 
as Exhibit 3 of this report. 



Figure 6: Total Value of Treasury Board Orders, 
2007/08-2011/12 ($ million) 

Source of data: Treasury Board 
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Figure 7: Total Value of Treasury Board Orders by 
Month Relating to the 2011/ 12 Fiscal Year 

Source of data: Treasury Board 



Figure 8: Authorized Transfers Relating to the Office of 
the Assembly, 2011/12 Fiscal Year ($) 

Source of data: Board of Internal Economy 





Authorized 


Month of Issue 


# 


($ million) 


April 2011-February 2012 


81 


2,544 


March 2012 


31 


329 


April 2012 


8 


54 


August 2012 


1 


190 


Total 


121 


3,117 



TRANSFERS AUTHORIZED BY THE 
BOARD OF INTERNAL ECONOMY 

When the Board of Internal Economy authorizes 
the transfer of money from one item of the Esti- 
mates of the Office of the Assembly to another item 
within the same vote, section 91 of the Legislative 
Assembly Act requires that we make special mention 
of the transfer (s) in our Annual Report. 

Accordingly Figure 8 shows the transfers 
made within Vote 201 with respect to the 2011/12 
Estimates. 

UNCOLLECTIBLE ACCOUNTS 

Under section 5 of the Financial Administration 
Act, the Lieutenant Governor in Council, on the 
recommendation of the Minister of Finance, may 
authorize an Order-in-Council to delete from the 
accounts any amounts due to the Crown that are 
the subject of a settlement or deemed uncollectible. 
The amounts deleted from the accounts during any 
fiscal year are to be reported in the Public Accounts. 

In the 2011/12 fiscal year, receivables of 
$816.4 million due to the Crown from individuals 
and non-government organizations were written 
off. (The comparable amount in 2010/11 was 
$432.1 million.) The writeoffs in the 2011/12 fiscal 
year related to the following: 



Item 2 Office of the Clerk 


(9,900) 


Item 3 Legislative Services 


(9,200) 


Item 6 Sergeant at Arms and Precinct 
Properties 


(41,800) 


Item 8 Caucus Support Services 


(109,900) 


Item 9 Members' Compensation and Travel 


(1,366,100) 


To: 


Item 4 Information and Technology Services 


51,100 


Item 5 Administrative Services 


7,300 


Item 10 Members' Office Support Services 


1,476,000 


Item 12 Lieutenant Governor's Suite 


2,500 



• $382.2 million for uncollectible retail sales tax 
($71.9 million in 2010/11); 

• $155.8 million for uncollectible corporate tax 
($65.1 million in 2010/11); 

• $114.1 million for uncollectible receiv- 
ables under the Student Support Program 
($145.2 million in 2010/11); 

• $86.3 million for uncollectible receivables 
under the Ontario Disability Support Program 
($118.8 million in 2010/11); 

• $48.9 million for uncollectible employer 
health tax ($6.4 million in 2010/11); and 

• $29.1 million for other tax and non-tax receiv- 
ables ($24.7 million in 2010/11). 

Volume 2 of the 2011/12 Public Accounts 
summarizes the writeoffs by ministry. Under the 
accounting policies followed in the preparation of 
the consolidated financial statements of the prov- 
ince, a provision for doubtful accounts is recorded 
against accounts receivable balances. Accordingly, 
most of the writeoffs had already been expensed 
in the government's consolidated financial state- 
ments. However, the actual writeoff in the accounts 
required Order-in-Council approval. 



Chapter 3 



Reports on 

Value-f or-money Audits 
and Reviews 



Our value-for-money (VFM) audits are intended to 
examine how well government ministries, organiza- 
tions in the broader public sector, agencies of the 
Crown and Crown-controlled corporations manage 
their programs and activities. These audits are 
conducted under subsection 12(2) of the Auditor 
General Act, which requires that the Office report 
on any cases observed where money was spent 
without due regard for economy and efficiency or 
where appropriate procedures were not in place to 
measure and report on the effectiveness of service 
delivery. Where relevant, such audits also encom- 
pass compliance issues. Essentially, VFM audits 
delve into the underlying operations of the ministry 
program or organization being audited to assess 
both their cost-effectiveness and the service level 
the public is receiving. This chapter contains the 
conclusions, observations and recommendations 
for the VFM audits and one review conducted in the 
past audit year. 

The ministry programs and activities and the 
organizations in the broader public sector audited 
this year were selected by the Office's senior man- 
agement on the basis of various criteria, such as 
a program's or organization's financial impact, its 
perceived significance to the Legislative Assembly, 
related issues of public sensitivity and safety, and 
the results of past audits and related follow-up 
work. 



We plan, perform and report on our value-for- 
money work in accordance with the professional 
standards for assurance engagements established 
by the Canadian Institute of Chartered Accountants, 
which encompass value for money and compliance. 
They entail conducting the tests and other proced- 
ures that we consider necessary, including obtaining 
advice from external experts when appropriate. 

Before beginning an audit, our staff conduct in- 
depth research into the area to be audited and meet 
with auditee representatives to discuss the focus 
of the audit, including our audit objectives and cri- 
teria. During the audit, staff maintain an ongoing 
dialogue with the auditee to review the progress of 
the audit and ensure open lines of communication. 
At the conclusion of the audit fieldwork, which is 
normally completed by late spring of that audit 
year, significant issues are discussed with the 
auditee and a draft audit report is prepared. Then 
senior Office staff meet with senior management 
from the auditee to discuss the draft report and the 
management responses to our recommendations. In 
the case of organizations in the broader public sec- 
tor, discussions are also held with senior manage- 
ment of the funding ministry. 

Once the content and responses for each VFM 
audit report are finalized, the VFM audit reports 
are incorporated as sections of this chapter of the 
Annual Report. 
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Chapter 3 
Section 




Cancer Care Ontario 

Cancer Screening 
Programs 




Cancer Care Ontario (CCO), formerly known as the 
Ontario Cancer Treatment and Research Founda- 
tion, was created in 1943 under the Ontario Cancer 
Treatment and Research Foundation Act. In 1997, the 
agency officially changed its name to Cancer Care 
Ontario and became governed by the Cancer Act. 
CCO is responsible for co-ordinating and overseeing 
cancer services in Ontario. It works with each of 
the province's Local Health Integration Networks 
(LHINs) to address local and regional needs and 
advises government on cancer matters. 

CCO directs health-care funding to hospitals 
and other care providers, with the aim of delivering 
quality and timely cancer services throughout the 
province. It is also responsible for implementing 
cancer prevention and screening programs. 

CCO has 13 regional cancer programs across the 
province. These programs bring together health- 
care professionals and organizations involved in 
cancer prevention and care. Each regional cancer 
program is led by a CCO regional vice-president. 
The regional programs are required to ensure that 
service providers meet the requirements and targets 
set out in their partnership agreements with CCO. 

Regional Cancer Centres are responsible for 
cancer screening and treatment services. Prior to 



2004, these centres were stand-alone organizations 
managed by CCO. On January 1, 2004, the centres 
formally integrated with their partner hospitals 
to provide more comprehensive care under 
one location. 

In 2011/12, CCO had total expenditures of 
$887 million, $92 million of which was spent on 
cancer screening programs. 



Audit Objective and Scope 



The objective of our audit was to assess whether 
Cancer Care Ontario (CCO) had adequate policies 
and procedures in place to: 

• monitor and assess whether cancer screening 
programs were provided in accordance with 
legislation, agreements and applicable direc- 
tives/policies; and 

• measure and report periodically on achieve- 
ments of cancer-screening-program objectives. 

Our audit objectives and criteria were reviewed 
and agreed to by CCO senior management. 

In conducting our audit, we reviewed relevant 
legislation and administrative policies and proced- 
ures, and we interviewed appropriate CCO head- 
office staff and Ministry of Health and Long-Term 
Care (Ministry) staff. We also visited sites in the 
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following regions: Greater Toronto Area, Hamilton 
Niagara Haldimand Brant, South West (London), 
Champlain (Ottawa) and North East (Sudbury) 
to review files and other summary information 
available for the Ontario Breast Screening Program 
and the colorectal cancer screening program. We 
also contacted these sites to obtain information on 
the cervical cancer screening program. To obtain a 
better understanding of and perspective on cancer 
screening programs, we spoke to various stakehold- 
ers, such as the Canadian Cancer Society, the Insti- 
tute of Clinical Evaluative Sciences, and the Cancer 
Quality Council of Ontario. We also contacted 
similar cancer agencies in Manitoba, Alberta and 
British Columbia. 

Our audit included a review of related activities 
of CCO's Internal Audit Department. We reviewed 
its recent reports and, when we planned our work, 
considered its audit work and any relevant issues it 
had identified. 



Summary 



Similarly to agencies in other jurisdictions such as 
Australia, New Zealand, the United Kingdom and 
other Canadian provinces, CCO has implemented 
cancer screening programs for breast, colorectal 
and cervical cancers. The key objective for each of 
the three cancer screening programs is to reduce 
the number of deaths from cancer through early 
detection and treatment. The mortality rates from 
these three types of cancer have fallen in Ontario 
over the past two decades. In this regard, Ontario's 
mortality rates are similar to the Canadian averages 
for these types of cancer. 

Our major observations with respect to these 
three screening programs were as follows: 
• We noted that CCO appropriately used rec- 
ognized clinical evidence in deciding what 
types of cancer warranted formal screening 
programs. Both the Ministry of Health and 
Long-Term Care, through a $45-million fund- 



ing commitment in 2010, and CCO, through 
recent initiatives, have clearly recognized 
the need to increase screening participation 
rates, especially for people considered to be at 
increased risk for cancer. 

• Participation in breast cancer and cervical 
cancer screening achieved ministry targets 
but fell short of CCO's own targets. As of 
2009/10, colorectal cancer screening in 
Ontario fell short of both the Ministry's and 
CCO's targets, and almost half of the targeted 
population remained unscreened. In total, 
between 2008 and 2010 only 27% of eligible 
women completed all three cancer screening 
tests recommended for their age group. As 
well, participation in the screening programs 
appears to have reached a plateau, and CCO is 
looking at ways to address this. 

• There were wait times for screening services 
for all three types of cancer: 

• In visits to regional offices, we found mam- 
mography screening wait times for women 
with an average risk for breast cancer but 
no symptoms ranged from just over two 
weeks to IOV2 months. CCO found that, in 
its program that targets women considered 
at high risk for breast cancer, the wait time 
for genetic assessments of screening eligi- 
bility averaged 84 days. 

• For colorectal screening, almost 30% of 
cases did not have a follow-up colonoscopy 
within the benchmark time established by 
CCO. The data showed that, in 2011/12, 
the median wait times for a colonoscopy 
were 12 weeks for individuals with family 
histories of colon cancer and six weeks for 
those needing to be followed up after a 
positive Fecal Occult Blood Test (FOBT) . 
However, we found instances in hospital 
records we reviewed where the wait times 
were as long as 72 weeks for individuals 
with family histories of colon cancer and 17 
weeks for those whose FOBT was positive. 



2012 Annual Report of the Office of the Auditor General of Ontario 



• For cervical cancer screening, a recent 
CCO preliminary review showed that the 
median wait time for a colposcopy (a diag- 
nostic procedure following up on abnormal 
cervical Pap test results) for high-grade 
abnormalities was about three months. 
Even though they are at greater risk of 
dying of cervical cancer, older women were 
screened at a much lower rate than younger 
women. CCO has said that physicians too 
often link Pap testing for cervical cancer to 
annual health exams, contraception counsel- 
ling and screening for sexually transmitted 
infections. Because older women often have 
fewer contraceptive and lifestyle reasons 
to see their doctor, they often do not get 
tested. Meanwhile, many low-risk younger 
women were being screened more often 
than necessary. 

The level of quality assurance measures for 
each of the screening programs varied con- 
siderably. CCO had developed a comprehen- 
sive quality assurance program to assess and 
monitor the breast cancer screening program, 
but 20% of screenings took place outside 
CCO's program and therefore were not subject 
to the requirements. CCO had set up some 
quality assurance processes for the colorectal 
cancer screening program, but none for the 
cervical cancer screening program. 
CCO did not analyze and monitor whether 
individual endoscopists (specialists who look 
inside a body cavity or organ using an endo- 
scope) met performance requirements. For 
instance, endoscopists are required to perform 
at least 200 colonoscopies annually to achieve 
or maintain competency. From data for the 
years 2008/09 to 2010/11, we found that 
more than 20% of endoscopists had not met 
this competency requirement. 
There was a significant backlog for follow-up 
reviews of mammography images in cases in 
which a woman was diagnosed with cancer 
after having had a breast cancer screening test 



that reported normal results. These follow- 
up reviews are done to see if the cancer was 
missed at the previous screening or whether 
the cancer actually developed after the 
previous screening (this is referred to as an 
interval cancer) . In 2009, 225 reviews were 
completed, of which 81 warranted further 
investigation. Of these 81 cases, about half 
were subsequently classified as missed-at- 
screening. No interval cancer reviews were 
done from the end of 2009 to July 2011, 
because CCO's Ontario Breast Screening 
Program radiologist-in-chief had retired and 
a new radiologist-in-chief was not hired until 
July 2011. At the time of our audit, a backlog 
of almost 900 interval cancer cases needed 
to be reviewed. CCO informed us that it 
expected to complete its follow-up review of 
these cases by December 2012. 
• CCO measured and reported on its achieve- 
ment of cancer-screening-program objectives, 
including annually publishing its program 
indicators for its three screening programs 
through the Cancer Quality Council of 
Ontario's Cancer System Quality Index and 
periodically issuing performance evaluations 
of its three cancer screening programs. 



Cancer Care Ontario (CCO) welcomes the rec- 
ommendations in this audit. The audit acknow- 
ledges the sound processes CCO has in place 
to assess whether a cancer screening program 
is needed and recognizes the Ontario-wide 
commitment to establish high-quality, evidence- 
based screening programs for breast, cervical 
and colorectal cancer. 

Screening is most effective when offered 
through a high-quality organized program 
that promotes participation; identifies and 
follows the target population through the 
screening journey; sends eligible people invita- 
tions, results letters and recalls to screening; 
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incorporates follow-up processes for those 
with abnormal test results; ensures access to 
high-quality diagnostic services; and includes 
program evaluation and reporting. 

The audit identifies areas that CCO is already 
working to address, reinforcing the value of its 
current work and its future directions. CCO will 
continue to work closely with the Ontario Min- 
istry of Health and Long-Term Care to ensure 
that Ontarians have access to high-quality 
cancer screening services. 



| OVERALL MINISTRY RESPON 




The Ministry acknowledges the recommenda- 
tions contained in the Auditor General of 
Ontario's cancer screening report and thanks 
him for conducting this timely audit. The 
Ministry is committed to the development and 
implementation of innovative initiatives and 
solutions that address the impact of cancer and 
other chronic diseases on Ontarians. We wel- 
come any insights and recommendations from 
the Auditor General that may help to further 
inform our ongoing planning and implementa- 
tion of cancer screening programs and services. 

Breast, cervical and colorectal cancer screen- 
ing saves lives when these cancers are detected 
in early stages. The Integrated Cancer Screening 
(ICS) program, delivered by the Ministry in 
partnership with CCO, is integrating existing 
breast, cervical and colorectal cancer screening 
programs and services into one co-ordinated 
provincial program to support the public, pro- 
viders and health-system planners in improving 
the quality and uptake of screening. 

CCO reports that between 1990 and 2007, 
breast cancer mortality rates declined by 35% 
for women aged 50-69 and by 29% for all ages. 
Since the program was launched in 1990, the 
Ontario Breast Screening Program (OBSP) has 
provided more than 4.1 million screens to more 
than 1.2 million women aged 50 and older 



across Ontario and detected more than 22,000 
cancers, the majority in the early stages. In 
March 2011, the OBSP was expanded to include 
women aged 30-69 years who are at high risk 
for breast cancer due to genetic factors, medical 
or family history. 

As for colorectal cancer screening, the 2012 
Cancer System Quality Index (CSQI) reported 
that in 2010, just over half of Ontarians aged 
50-74 were up-to-date with Fecal Occult Blood 
Test, flexible sigmoidoscopy and/or colonos- 
copy. The Ministry and CCO are committed to 
further increasing participation in colorectal 
cancer screening, including evaluating new 
screening technology for use in Ontario. 



Detailed Audit Observations 



STRATEGIC INITIATIVES 

Over the years, Cancer Care Ontario (CCO) has 
developed a number of strategic initiatives to 
reduce the incidence and mortality from cancers. In 
2003, CCO in collaboration with the Canadian Can- 
cer Society issued Cancer 2020, Targeting Cancer: 
An Action Plan for Cancer Prevention and Detection, 
a report that provided a long-term provincial plan 
for reducing the number of people diagnosed with, 
and dying from, cancer by 2020. In addition to this 
long-term plan, CCO developed and released three 
separate three-year Ontario Cancer Plans to provide 
a more detailed road map for cancer care. 

In 2010, the Ministry of Health and Long-Term 
Care (Ministry) initiated an Integrated Cancer 
Screening (ICS) strategy. The Ministry committed 
$45 million to CCO from the 2010/11 through 
2012/13 fiscal years to implement the strategy, 
which aims to, among other things, increase patient 
participation in screening, make primary-care 
providers aware of their role in the process, expand 
information systems to better identify eligible and 
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high-risk people, and better monitor and report on 
the screening process. 

CCO has implemented three cancer screening 
programs — the Ontario Breast Screening Program, 
ColonCancerCheck and a cervical cancer screen- 
ing program. Our review showed that CCO had 
sound processes in place to assess whether a cancer 
screening program should be established and 
that its decisions were based on clinical evidence 
that demonstrated that screening was effective in 
reducing mortality. According to a national body's 
expert panel, Canadian Partnership Against Cancer 
(CPAC), a "reduction in cancer mortality is the 
definitive requirement to confirm that the screening 
test is effective." 

We noted that the criteria used to determine 
which types of cancer warranted screening pro- 
grams were in accordance with principles estab- 
lished by the World Health Organization (WHO) . 
These criteria included, for example, an assessment 
of whether the condition is an important health 
problem, whether a suitable screening test or 
examination exists, and whether the overall benefit 
of the screening program outweighs potential harm 
from its application. For instance, in the case of 
prostate cancer, the balance of evidence has sug- 
gested that the harm of implementing an organized 
prostate cancer screening program may outweigh 
the benefits. CCO has indicated that it is continuing 
to monitor the current research work in this and 
other types of cancer, such as lung cancers. 

CANCER SCREENING PROGRAMS 

There is substantial evidence to demonstrate that 
early screening and detection is critical in helping 
to reduce deaths from cancer. Early detection of 
cancers can lead to less invasive treatments and 
improved health outcomes. 

As agencies in other jurisdictions such as 
Australia, New Zealand, the United Kingdom and 
other Canadian provinces have done, CCO has 
implemented cancer screening programs for breast, 
colorectal and cervical cancers. 



A review of the mortality rates from 1992 to 
2012 showed a decrease in the rates for each of the 
three types of cancers over the period. However, 
the rates of decline have slowed in recent years, as 
shown in Figure 1. According to Canadian Cancer 
Society data, Ontario's mortality rates for these 
three cancer types are comparable to those of most 
Canadian provinces, with British Columbia, Alberta 
and New Brunswick having slightly lower rates. 

CCO told us that its efforts under the Integrated 
Cancer Screening strategy to improve screening 
participation rates and the quality of screening ser- 
vices would help accelerate the reduction of cancer 
mortality rates. 

Participation in and Access to Cancer 
Screening Programs 

Screening programs are effective if they reach a 
sizeable percentage of the target population. Over 
the years, many different target participation rates 
have been established to guide cancer screening. 

The Cancer 2020 report in 2003 set participation 
rate targets of 90% to 95%, but CCO subsequently 
established what it deemed were more realistic 
target rates in its three-year Ontario Cancer Plans. 
Then, the Ministry, seeing that participation rates 
for all three programs had been levelling off, 
worked with CCO to develop an Integrated Cancer 
Screening (ICS) strategy to improve participation 
in screening for breast, cervical and colorectal 
cancers. The strategy included targets for gradually 
increasing participation rates from 2009/10 to 
2013/14. For example, the breast cancer screening 
target increases from 66% in 2009/10 to 73% in 
2013/14. 

Participation target rates apply only to people 
deemed to be eligible for the screening. Eligibility 
criteria are based on such things as an age range or 
a person's other risk factors in developing a particu- 
lar cancer, such as family history. At the time of our 
audit, the most recent CCO participation rate data 
available was for 2009/10, as shown in Figure 2. 
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Figure 1: Mortality Rates for Breast, Colorectal and Cervical Cancer in Ontario (per 100,000 Ontarians) 

Source of data: Cancer Care Ontario 
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* Breast and cervical cancer mortality rates are per 100,000 women 
in Ontario. 

a. Cervical cancer mortality rates for 2008 to 2012 are not available. 

b. Breast and colorectal cancer mortality rates for 2009 to 2012 
are estimated. 



To promote public awareness of the cancer 
screening programs, the Ministry spent a total of 
$13 million from 2007/08 to 2011/12 for com- 
munication and promotion; CCO spent $5.9 mil- 
lion over the same period to promote the screening 
programs to various health-care providers and 
stakeholders. 

The Ministry and CCO were jointly responsible 
for the promotion of the cancer screening pro- 
grams. An overall framework was developed to 
ensure a consistent and focused message, with the 
Ministry leading public communications and CCO 
leading provider education. The Ministry and CCO 
worked together on certain specific initiatives. 

Our specific observations with respect to partici- 
pation in and access to each of the cancer screening 
programs are as follows. 



Figure 2: Target and Actual Participation Rates in 
Cancer Screening Programs (%) 

Source of data: Cancer Care Ontario 
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Breast Cancer Screening 

The Ontario Breast Screening Program (OBSP) was 
implemented in 1990, and the eligible population 
was women aged 50 to 69 years (changed to 50 to 
74 years in November 2011) with average risk of 
developing breast cancer. Average risk means there 
is no risk factor other than being a woman and 
being older. 
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In July 2011, Ontario became the only jurisdic- 
tion in Canada to integrate screening for women at 
high risk for breast cancer into an organized breast 
cancer screening program. The eligible population 
is women aged 30 to 69 who are at high risk of 
developing breast cancer. High-risk factors include 
having a specific genetic mutation, a family history 
that suggests hereditary breast cancer, a 25% or 
greater lifetime risk confirmed through genetic 
assessment, and having had radiation therapy to 
the chest before age 30 or more than eight years 
ago as treatment for another cancer or condition. 

The average-risk program has tended to meet its 
participation targets, while the high-risk program 
was well below its target, although it should be 
acknowledged that the program is relatively new. 

Breast cancer screenings and assessments are 
provided by more than 150 OBSP affiliate sites in 
hospitals and independent health facilities. About 
523,000 women were screened through the OBSP 
in 2011/12. (Women can be screened outside of 
this program. In the 2010/11 fiscal year, the most 
recent year for which figures are available, about 
112,800 women received mammogram services 
outside of the OBSP in independent health facili- 
ties.) About $78 million was allocated to OBSP for 
breast cancer screenings. 

For women with average risk of breast cancer, 
guidelines recommend cancer screening with mam- 
mography every two years. For women at high risk, 
the guidelines recommend annual screening with 
magnetic resonance imaging (MRI) or ultrasound, 
as well as mammogram. 

In 2009/10, the most recent year for which 
statistics are provided, the participation rate for the 
eligible population at average risk of breast cancer 
was 66.8%, which meets the Integrated Cancer 
Screening strategy target, but is short of the CCO's 
target of 70% in its Ontario Cancer Plans. 

In 2011/12, the Ministry allocated about 
$11.6 million for screening of high-risk women. 
Of this, approximately $6.5 million was provided 
to CCO, including $4.7 million for an expected 
20,000 screening exams and associated costs, and 
$1.8 million for genetic assessments. 



To access genetic assessment services, a woman 
must be referred to the OBSP by her doctor. Women 
confirmed as being at high risk for breast cancer 
will be booked for both a breast MRI and mammog- 
raphy. A May 2011 announcement indicated that 
screening these women with annual breast MRI 
and mammography will detect approximately 17 
cancers per year in every 1,000 women screened. 

Program data for the first nine months of the 
program, from July 1, 2011, to March 31, 2012, 
showed that about 5,000 women aged 29 to 69 
were referred to the OBSP high-risk program. 
Of this total, about 600 women were screened. 
This number is well below the 20,000 that the 
Ministry expected for the program. CCO stated 
that the low number could be because the program 
started in the summer, or because women first 
had to get a referral from their family doctor to 
the program and many of them then had to have a 
genetic assessment. Of the $4.7 million allocated 
for high-risk screenings and related services, CCO 
identified $3.3 million that needed to be returned 
to the Ministry after the end of the fiscal year. CCO 
told us it has lowered its projection for 2012/13 to 
5,000 screens. 

Wait Times for Breast Cancer Screening Services 

Each OBSP affiliate site or regional office manages 
its own mammography screening bookings. The 
OBSP does not have a standard wait time for such 
services. At the three regional offices we visited, we 
found variations in the wait times ranging from just 
over two weeks to IOV2 months. One reason that 
women in the program wait could be due to the 
high number of screenings and assessments done at 
these sites for women who are not eligible for the 
breast screening program. One site we visited did 
20,500 mammograms in the 2010/11 fiscal year, 
of which only 9,400 were for those women defined 
as being eligible under the OBSP criteria. CCO has 
data on the provincial totals of non-OBSP screens 
conducted, but it did not collect such data on a 
site-by-site basis to help it assess the capacity of its 
more-than-150 breast cancer screening facilities. 
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In addition, data from 2008 (the most recent 
year for which data is available) shows that 56% of 
abnormalities identified in mammograms as requir- 
ing biopsy were followed up within seven weeks. 
This is below the national target established by the 
Public Health Agency of Canada of 90% within that 
time frame. CCO's comparison with other jurisdic- 
tions found that no other province had achieved this 
national target. Public Health Agency of Canada data 
from 2005/06 shows that only Nova Scotia did better 
than Ontario on this measure, with 58% of instances 
followed up on time, as compared to 57% in Ontario. 

On our regional visits, we found there were 
wait times associated with various stages of the 
screening for those considered at high risk for 
breast cancer — up to six months at some sites. CCO 
conducted a survey at the end of 2011 of all genetic 
assessment clinics that determined eligibility for 
high-risk breast cancer screening. The survey found 
that wait times from when a woman received her 
referral to her first appointment averaged 84 days. 
Most clinics in the survey indicated that the waits 
were primarily a result of their not having enough 
staff to do the genetic assessments more quickly. 



RECOMMENDATION 1 



To improve breast cancer screening services to 
eligible participants, especially those considered 
to be at high risk of breast cancer, Cancer Care 
Ontario (CCO) should periodically evaluate 
the wait times at each of its screening facilities. 
As well, CCO should take measures to increase 
its capacity to expedite genetic assessments for 
women who have been referred to the high-risk 
program by their doctors. 




CCO agrees that a reduction in wait times at all 
screening sites is important in order to improve 
breast cancer screening and assessment servi- 
ces. CCO will work with the regions to evaluate 
and improve wait times for screening and 
follow-up of abnormal screens. 



CCO is conducting a one-year evaluation to 
identify areas where improvements can be made 
to the Ontario Breast Screening Program's new 
high-risk program. This is to be completed by 
March 2013. This evaluation will include wait 
times for MRIs and genetic assessments (i.e., 
counselling and testing) for women in the high- 
risk screening program. Based on the outcomes 
of the one-year evaluation, CCO will implement 
improvements to the program, with a special 
focus on reducing wait times for MRIs and 
breast assessments, where needed, beginning in 
2013/14. 

| MINISTRY RESPONSE 

The Ministry supports the evaluation of wait 
times for breast screening at each of its screen- 
ing sites. The Ministry will work with CCO to 
evaluate and improve wait times for screening 
and follow-up of abnormal screens at all screen- 
ing sites. 

The Ministry has provided CCO with resour- 
ces to conduct an evaluation of the Ontario 
Breast Screening Program to identify improve- 
ments, with a special focus on reducing wait 
times for MRIs and assessments as appropriate. 



Colorectal Cancer Screening 

In 2007, the Ministry committed $195 million 
for colorectal cancer screening over the five-year 
period from April 1, 2007, to March 31, 2012. CCO 
then launched the ColonCancerCheck program 
with total committed funding of $72 million for the 
same five-year period. The eligible population of 
the program was men and women aged 50 to 74. 

The program chose the guaiac Fecal Occult 
Blood Test (FOBT) kit as the primary colorectal 
cancer screening tool for Ontarians considered to 
be at average risk. This would include those with 
no family history of colorectal cancer or symptoms. 
FOBT testing is recommended every two years. 



2012 Annual Report of the Office of the Auditor General of Ontario 



When an FOBT is positive, the primary-care pro- 
vider should refer the patient for a colonoscopy 
follow-up. 

The program also includes people considered to 
be at increased risk — these are people with one or 
more immediate relatives who have, or have had, 
colorectal cancer — but the protocol is different. 
People at increased risk can bypass the FOBT and 
are fast-tracked to have a colonoscopy. If no cancer 
is detected during the colonoscopy, a person is 
encouraged to be tested again every five to 10 years. 

In 2011/12, 63 hospitals signed agreements 
with CCO and received incentive funding of 
$4.6 million to participate in the ColonCancer- 
Check program. The hospitals performed a total of 
14,300 colonoscopies in that year. 

Participation in Colorectal Cancer Screening 

ColonCancerCheck reports that the participation 
rate among the eligible population (men and women 
aged 50 to 74) in 2009/10 was 27.4%, short of the 
ICS target of 32% and CCO's Ontario Cancer Plan 
target of 40%. However, the program measured 
only the rate of participation in the FOBT. In our 
visits to regional offices, and to some hospitals in 
these regions, we were told that the low rates might 
be pardy due to the fact that many doctors think 
the FOBT is not reliable enough as a screening tool, 
and they were instead referring many average-risk 
people directly for colonoscopy screening. Indeed, 
a Ministry-commissioned survey in 2010 found 
that 37% of physicians believed the FOBT was not 
reliable enough to be used as a population-based 
screening tool. Accordingly, there were likely many 
more average-risk people being screened than were 
participating in the FOBT. (CCO told us at the time of 
our audit that it was conducting a pilot project using 
another stool-based test kit, the Fecal Immunochem- 
ical Test, or FIT, which has a higher sensitivity than 
the approved FOBT kit being used to that point; the 
different kit may address physicians' concerns about 
using the FOBT to screen for colorectal cancer.) 



As well, because the participation rate figure 
did not include the direct colonoscopy screening of 
people considered at increased risk for colorectal 
cancer, the overall participation in colorectal cancer 
screening in Ontario was understated. 

In December 2011, a Joint Steering Committee 
of the Ministry and CCO approved a method of 
calculating the colorectal cancer screening rate that 
includes people screened with FOBT, flexible sig- 
moidoscopy or colonoscopy, and decided that this 
calculation would be reported in addition to the 
FOBT participation rate. However, this method of 
calculation will result in overstating the screening 
participation rate because it also includes all indi- 
viduals receiving treatment for colorectal cancer 
or precancerous lesions, and such treatment is not 
considered to be screening. 

Under this new calculation method, the colo- 
rectal testing rate was reported in 2012 as 53% 
for 2010. It included everyone who had completed 
an FOBT in the previous two years, or who had 
had a flexible sigmoidoscopy in the previous five 
years, or colonoscopy in the previous 10 years, 
from all sources, including private colonoscopy 
clinics. Thus, about half of the eligible population 
remained unscreened. 

Colonoscopy Wait Times at Hospitals 

When the colorectal cancer screening program 
commenced in 2007/08, 57 hospitals participated 
in the program. This increased to 74 in the 2008/09 
fiscal year and then dropped to 64 in 2010/11 and 
62 in 2011/12. 

The Canadian Association of Gastroenterology 
set a goal of two months to complete a follow- 
up procedure for a positive FOBT, a benchmark 
adapted by CCO as eight weeks. For people with a 
family history of colorectal cancer, the benchmark 
for the time from referral to colonoscopy was set 
by the Association as six months, adapted by CCO 
as 26 weeks. The ColonCancerCheck program 
set a provincial target of 75% for the eight-week 
benchmark and a provincial target of 80% for the 
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26-week benchmark for those with a family history 
of colorectal cancer. 

The Cancer System Quality Index (CSQI), 
recently published by the advisory group Cancer 
Quality Council of Ontario, reported that in 2011 
73% of positive FOBT cases met the eight-week 
wait-time benchmark, an improvement over 60% 
in 2009. As stated in CCO reports, 80% of family 
history cases met the 26-week benchmark in 
2010/11, an improvement over 71% in 2008/09. 
Our review of records at hospitals found that wait 
times exceeded the benchmarks of eight weeks for 
positive FOBT follow-up and 26 weeks for family 
history cases. Our review identified instances where 
individuals with positive FOBTs waited as long as 
17 weeks for a follow-up colonoscopy and those 
with family history of colon cancer waited as long 
as 72 weeks for a colonoscopy. 

While there was improvement in the two wait- 
time rates, almost 30% of participants did not 
receive a follow-up colonoscopy within eight weeks 
of a positive FOBT result and within 26 weeks of 
a referral for increased risk. Some people were 
screened at private clinics, but CCO did not have 
access to the referral dates to private clinics to 
assess the wait times of participants. 

CCO attributed the shortfall in meeting its 
targets to a number of things, including physicians 
who did not follow up with participants, reluctance 
of some people to have a colonoscopy, and phys- 
icians who told patients with positive FOBT results 
to repeat the test when they should have been 
referred at that point for a colonoscopy. 



RECOMMENDATION 2 



To increase participation and improve its 
colon cancer screening efforts, Cancer Care 
Ontario should: 

• examine and work to address the concerns 
doctors have with the effectiveness of the 
Fecal Occult Blood Test as a screening 
tool; and 



• explore approaches for reducing the wait 
times for colonoscopy procedures, especially 
those for increased-risk patients. 

| CCO RESPONSE 

CCO agrees with this recommendation and is 
working to increase participation and improve 
the colorectal cancer screening program. It will 
continue to educate primary-care providers 
about the highest-quality evidence supporting 
FOBT screening. In addition, it will continue 
to evaluate the feasibility of introducing other 
screening tests that have recently been shown 
by highest-quality evidence to reduce colorectal 
cancer mortality — namely, a more sensitive 
stool-based test (the Fecal Immunochemical 
Test, or FIT) and flexible sigmoidoscopy. Both 
methods are appropriate for average-risk 
screening and may be more acceptable to 
primary-care providers, including doctors. 

CCO will continue to work with the regions 
to improve wait times for colonoscopy at sites 
that are not meeting wait-time targets, espe- 
cially for increased-risk patients (those with a 
first-degree family history of colorectal cancer), 
through regular quarterly performance reviews 
and contract management. 

MINISTRY RESPONSE 

The Ministry has provided CCO with funding to 
conduct provider education and a pilot evalua- 
tion of newer Fecal Immunochemical Test tech- 
nology for use in Ontario. 

The Ministry has also provided CCO with 
funding to: 

• expand colonoscopy capacity to reduce wait 
times for individuals who are at increased 
risk of colorectal cancer; and 

• conduct a pilot to leverage non-hospital 
colonoscopy clinics to improve capacity for 
colonoscopy services, as part of the Colon- 
CancerCheck program. 
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Cervical Cancer Screening 

The cervical cancer screening program was 
launched on June 15, 2000. In 2010/11, the Min- 
istry paid $54.8 million to doctors and labs for Pap 
tests. Screening is primarily performed with Pap 
tests done by physicians in their offices as part of 
routine checkups. When a low-grade abnormality 
shows up on a Pap test, the woman usually receives 
a repeat Pap test in six months. When a high-grade 
abnormality shows up on a Pap test, the woman is 
usually referred for a colposcopy — a visual exam- 
ination of the cervix using an instrument called a 
colposcope. In some cases, tissue is removed in a 
biopsy, and a pathologist makes a diagnosis. Col- 
poscopies are performed in hospital-based clinics or 
in private clinics. 

CCO has recently issued updated guide- 
lines that recommend the use of the Human 
Papillomavirus (HPV)-DNA test as a cervical cancer 
screening tool for women aged 30 years and older. 
The agency is working with the Ministry to explore 
how the test can be incorporated into the screen- 
ing program. 

Participation Rates of Women with Invasive 
Cervical Cancer 

Cervical cancer is largely preventable with HPV 
immunization, regular screening and appropriate, 
timely follow-up of abnormal results. Cervical 
cancer mortality increases steeply from age 45. The 
most recent Ontario Cancer Registry data, from 
2009 to 2011, shows that 83% of deaths from cer- 
vical cancer occurred in women over 45. 

Our review showed that in spite of the fact 
that older women were at increased risk of dying 
from cervical cancer, they were not appropriately 
targeted for screening and were inadequately 
screened. For instance, between 2009 and 2011, 
older women were twice as likely not to have cer- 
vical cancer screening in the three years prior to 
being diagnosed with invasive cervical cancer than 
younger women. Specifically, two-thirds of women 
aged 45 to 74 years diagnosed with invasive 



cervical cancer did not have cervical cancer screen- 
ing in the three years prior to being diagnosed, 
compared to only one-third of women aged 21 to 
34 years that were not screened. In fact, half of 
women aged 55 to 74 years were not screened in 
the 10 years prior to being diagnosed with invasive 
cervical cancer. 

Frequency of Cervical Cancer Screening 

In general, CCO recommends cervical cancer 
screening every three years for all women aged 21 
to 69 who are or have ever been sexually active. 
However, if a woman receives an abnormal test 
result, CCO recommends that she be tested annu- 
ally until she has three successive normal results. 

The overall provincial participation rate for cer- 
vical cancer screening in 2009/10 was 72%, which 
fell short of the Ontario Cancer Plan target of 85%, 
but met the Integrated Cancer Screening target. 
There was a significant difference in participation 
rates among age groups. CCO's program evaluation 
reported that the highest rates of screening par- 
ticipation were among women aged 20 to 29 years 
(74%), and the lowest rates were among women 
aged 60 to 69 years (66%.) Accordingly, younger 
women, who have a lower risk of cervical cancer, 
have the highest rates of annual Pap test screen- 
ing. Our review showed that 16% of women aged 
20 to 29 who had normal Pap test results in 2009 
were screened again within 12 months. This only 
occurred in 7% of women aged 70 and older. 

Discussions with CCO management indicated 
that younger women (20 to 34 years) are more 
frequently screened and rescreened because 
physicians often link Pap testing to annual health 
exams, contraception counselling and screening 
for sexually transmitted infections. The Ministry 
had identified this as an issue in 1996. Because 
older women often have fewer contraceptive and 
lifestyle reasons to see their doctor, they do not 
get tested often enough, if at all, even though they 
are at a greater risk of developing and dying from 
cervical cancer. 



Cancer Screening Programs 



CCO officials said the issue of too much or too 
little screening can also be attributed to lack of an 
organized system for telling people when they are 
due to have the test. For the cervical cancer screen- 
ing program to have an organized call and recall 
system, it must be allowed to collect health infor- 
mation about individuals without their consent. 
This requires being approved for Prescribed Regis- 
try status by the Ministry and the Information and 
Privacy Commissioner. CCO obtained Prescribed 
Registry status for the Ontario Cancer Screening 
Registry in May 2011. The Information and Privacy 
Commissioner approved CCO's information prac- 
tices and procedures in respect of the Registry in 
October 2011. As a result, the program can now col- 
lect data to identify eligible women and send them 
directly all appropriate correspondence about test 
results and to invite or recall them for screening. 

According to CCO, it would be a better practice 
to encourage doctors to view the Pap test as a 
separate service, and not tie it to appointments for 
contraception counselling, annual health exams 
and testing for sexually transmitted infections. 
CCO recommended that the Ministry, through its 
negotiations with the Ontario Medical Association, 
align its incentive payments for physicians with 
its updated cervical cancer screening guidelines, 
which were released to the public and health-care 
providers in May 2012. The current physician 
incentive bonus has been based on the percentage 
of the target population who received a Pap test 
in the 30 months prior to March 31 of the fiscal 
year for which the bonus is being claimed. This is a 
six-month shorter interval than the 36 months that 
the CCO recommends between Pap tests, and may 
encourage over-screening. In addition, there were 
no financial disincentives to screening women more 
frequently than at the recommended intervals. 

Wait Times for Colposcopy Services 

For screening to be effective, timely follow-up of 
abnormal Pap test results is critical. The 2012 Can- 
cer System Quality Index noted that 17% of women 



aged 20 to 69 did not have a follow-up colposcopy 
within six months of a high-grade abnormal Pap 
test. The 2008 colposcopy guidelines specify that 
less severe cytological findings should be followed 
up with colposcopy within eight to 12 weeks, while 
more severe findings should be followed up within 
a shorter time. As a minimum standard, time from 
referral to colposcopy should not exceed six months. 

During our audit, we found that CCO had not 
sent out any correspondence to the affected individ- 
uals, but CCO informed us that now that it has Pre- 
scribed Registry status, it planned, as of fall 2012, to 
send result letters to all women aged 21 to 69 years 
with abnormal or unsatisfactory Pap test results. 

In February 2012, CCO did a preliminary review 
of colposcopy data for the years 2008 to 2010 to 
determine the median time for receiving a col- 
poscopy. This preliminary review showed that the 
median wait time for high-grade abnormalities was 
generally about three months. 



RECOMMENDATION 3 



To improve the effectiveness of its cervical 
cancer screening services, Cancer Care 
Ontario should: 

• target promotional and educational efforts 
to increase participation and rescreening 
rates among older women; 

• educate the public and health-care providers 
on appropriate cervical cancer screening 
intervals; and 

• monitor wait times for colposcopy proced- 
ures for timely follow-up of women with 
abnormal Pap test results. 



| CCO RESPONSE 

CCO agrees with this recommendation and 
is actively working to strengthen the cervical 
cancer screening program. Providers who 
perform cervical screening are key to ensuring 
that screening and follow-up are done according 
to guidelines. In May 2012, CCO disseminated 
its new cervical cancer screening guidelines to 
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providers across Ontario, along with information 
they can distribute to women. 

In addition, efforts to increase awareness 
of cervical cancer screening guidelines among 
providers and women of all ages will continue 
through targeted media outreach and corres- 
pondence campaigns beginning in July 2012. 
This includes the development of awareness 
campaigns with regional partners to promote 
appropriate screening for under-/never- 
screened populations and older women. 

CCO will work with hospitals and colposcopy 
clinics to monitor and improve colposcopy wait 
times, where needed, and will work with primary- 
care providers to improve follow-up rates for all 
women with abnormal Pap test results. 



| MINISTRY RESPONSE 




The Ministry will continue to work with and 
fund CCO to identify effective strategies for 
delivery of promotion and education aimed at 
increasing knowledge and awareness among the 
public and health-care providers on appropriate 
cervical cancer screening and clinically recom- 
mended guidelines. 



Cancer Screening for People with No 
Family Physicians 

According to the Ministry, about 6% of the adult 
population does not have a family physician. Cur- 
rently, when anyone without a primary-health-care 
provider participates in any of the three cancer 
screening programs, CCO is responsible for ensur- 
ing that abnormal test results are followed up. CCO 
then encourages these people to contact Health 
Care Connect to find a physician for ongoing pri- 
mary care. However, this process does not actively 
seek out people who do not have family physicians 
and get them into screening programs. 

CCO has recognized that people who never, 
or rarely, participate in screening programs often 



face challenges relating to low incomes, immigrant 
backgrounds, functional difficulties or sexual 
orientation. CCO indicated that "current research 
suggests that local, customized interventions best 
address the specific barriers to screening experi- 
enced by this group." As a result, CCO received a 
commitment from the Ministry of $4.5 million for 
2010/11 to 2012/13 to develop initiatives directed 
to the under- or never-screened population. 

In 2010/11, CCO selected five of its 13 regions 
for projects for under- and never-screened people, 
and it is providing funding to these five regions 
for two years, after which time it will evaluate the 
initiatives to see if they should be expanded. CCO 
informed us that there are plans to perform more 
detailed analysis on a LHIN level to expand on the 
current regional projects in the 2013/14 fiscal year. 

In addition, under the Integrated Cancer Screen- 
ing strategy, CCO has various proposals or pilot 
projects to help improve participation in screening 
programs among people who do not have family 
physicians. Such initiatives include sending colo- 
rectal cancer screening kits directly to a sample 
of eligible people, providing screening invitations 
and reminders to eligible people, and developing 
community-based education/recruitment to engage 
under-screened populations. 



RECOMMENDATION 4 



The Ministry of Health and Long-Term Care 
should monitor and assess current Cancer Care 
Ontario initiatives designed to improve partici- 
pation in screening programs among people 
who do not have family physicians to gauge 
their effectiveness. 




Improving participation in screening programs is 
a key objective of the Integrated Cancer Screen- 
ing program. The Ministry provides CCO with 
the appropriate resources, mandate and public 
support to carry out its initiatives and activities. 



The Ministry and CCO are finalizing the cur- 
rent Transfer Payment Accountability Agreement, 
which includes appropriate mechanisms for mon- 
itoring, oversight and reporting of CCO's activities 
against clearly defined objectives and targets. 

In addition, the Ministry's Health Care Con- 
nect program continues to connect unattached 
people with primary-care providers. Since its 
inception, Health Care Connect has matched 
and referred more than 100,000 people. 

| CCO RESPONSE 

CCO agrees with this recommendation 
and will work with the Ministry to ensure 
appropriate monitoring of CCO's activities to 
improve participation among people who do 
not have primary-care providers. CCO and 
the Ministry are working to determine how to 
increase screening access for all eligible people, 
including those without providers, such as by 
allowing self -referral for screening, having other 
health-care professionals provide screening 
tests, mailing FOBT kits directly, and providing 
screening in mobile settings. CCO and the 
Ministry are collaborating to make necessary 
regulatory changes to expand access to screening 
through these channels, such as by permitting 
laboratories to process screening tests dispensed 
by non-physicians. 

Increasing screening participation for people 
without primary-care providers must be coupled 
with ensuring timely follow-up for those who 
have abnormal test results. CCO is working 
with the Ministry to implement an organized 
follow-up model for all those who have been 
screened, particularly people without primary- 
care providers. 

MONITORING FOR QUALITY OF SERVICES 

Monitoring the quality of cancer screening pro- 
grams is important to help ensure that the pro- 



Cancer Screening Programs 

grams live up to established minimum standards 
and that a process is in place to assess their reliabil- 
ity on an ongoing basis. Patients and their doctors 
need to be able to trust the results. Our review of 
the quality assurance programs in place for each 
of the screening programs found significant varia- 
tions. CCO had developed a comprehensive quality 
assurance program for monitoring the breast cancer 
screening program, but it had established only lim- 
ited monitoring for the colorectal cancer screening 
program, and none for the cervical cancer screen- 
ing program. As well, our research indicated that 
other jurisdictions, such as the United Kingdom, 
have well-defined quality assurance processes that 
may warrant consideration here in Ontario. 

Breast Cancer Screening 

CCO had developed a comprehensive quality assur- 
ance program for monitoring the Ontario Breast 
Screening Program (OBSP). 

Sites must meet specific minimum requirements 
before they can participate in the OBSP. Among 
other things, they must receive and maintain 
accreditation from the Canadian Association of 
Radiologists Mammography Accreditation Program 
(CAR-MAP). The accreditation covers radiologist 
and medical radiation technologist qualifications, 
equipment, quality control, quality assurance, 
image quality and radiation dose. 

However, in 2009/10, 20% of breast cancer 
screenings were done outside of the OBSP and were 
performed at non-OBSP sites that were not subject 
to the monitoring requirements. In discussions at 
regional offices we visited, we were told that the 
use of non-OBSP sites could be because doctors 
were referring women to sites that were close by 
or even in the same building as the doctor's office, 
or because doctors and patients are not necessarily 
aware that there is a difference between OBSP 
and non-OBSP services. As a consequence, these 
women did not have access to CCO's follow-up of 
abnormal test results, reminders or recalls for the 
next appointment, or to the CCO's quality assurance 
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processes as did women screened at OBSP sites, 
although their results were still read by a radiologist 
and sent to their doctor for follow-up. 

We were informed by CCO during the audit 
that the College of Physicians and Surgeons of 
Ontario (College) would be requiring CAR-MAP 
accreditation for non-OBSP independent health 
facilities by early 2013. The College has since 
included a requirement for CAR-MAP accreditation 
by 2014 in the updated Clinical Practice Parameters 
and Facility Standards for all independent health 
facilities. Hospitals that provide mammography 
services are not required to have CAR-MAP 
accreditation, but CCO has recommended that the 
Ministry require it. The Ministry has agreed that 
CCO should strike an expert panel and work with 
CAR-MAP and the College to further develop this 
recommendation. Subsequent to our audit, we were 
informed that the Ministry was reviewing options 
to ensure that all sites providing mammography 
services are CAR-MAP-accredited by 2014. 

In addition to requiring accreditation, the OBSP 
has established several quality assurance processes. 
These include regular reviews of the work of the 
radiologists who assess the screens, inspections of 
mammography machines every six months, reviews 
of the work of medical radiation technologists, and 
chart audits to ensure information on participants 
is complete and up to date. 

For quality assurance purposes, the OBSP also 
conducts what are called "interval cancer reviews" 
of cases in which a woman has been diagnosed 
with cancer after having had a previous screen- 
ing test that reported normal results. This is to 
determine if the cancer was missed at the previous 
screening or whether the cancer developed subse- 
quent to the screening, which provides feedback to 
OBSP radiologists. 

In 2009, 225 such cases had been reviewed and 
81 were further investigated. Of these 81 cases, 
42 were subsequently classified as missed-at- 
screening. No interval cancer reviews were done 
from the end of 2009 to July 2011, because CCO's 
OBSP radiologist-in-chief had retired and a new 



radiologist-in-chief was not hired until July 2011. 
At the time of our audit, a backlog of almost 900 
interval cancer cases needed to be reviewed. CCO 
informed us that it expected to complete its follow- 
up review of these cases by December 2012. 

CCO arranges for independent inspection of 
mammography machines every six months to make 
sure their radiation levels are within the acceptable 
range. We reviewed a sample of inspection reports 
at three regions we visited. One region had not 
received all the reports from its sites on how they 
followed up on any issues that arose in the inspec- 
tions. Some of these reports were due in August, 
September or October of 2011. At another region, all 
issues that were reported had been addressed, and 
at the third, no significant issues were identified. 

Regional offices are required to conduct audits 
of patient records (chart audits) to ensure that 
data entered in the screening system is accurate, in 
accordance with OBSP standards and policies, and 
consistent with data entered in the provincial infor- 
mation system. We found that the chart audit policy 
did not specify the sample sizes to be reviewed, the 
frequency of the reviews, and, when concerns are 
identified, what the subsequent review frequency 
must be. At the regions we visited, we found signifi- 
cant variations: 

• Only three of the four regions we visited 
conducted chart audits. The region that did 
not perform chart audits chose to conduct 
sample reviews for only cases with abnormal 
screen results. 

• For the three regions that conducted chart 
audits, the frequency and types of reviews 
varied. One region conducted chart audits 
every three years with less extensive chart 
reviews between the audit years. The other 
two regions conducted annual chart audits. 

• The three regions each reviewed a sample of 
20 files for each site they audited, regardless of 
the number of yearly screens performed at the 
site. Therefore, a site that performed 10,000 
screens annually had the same sample size as a 
site that performed 500 screens annually. 



Cancer Screening Programs 



Colorectal Cancer Screening 

CCO and the Ministry maintain some monitoring 
mechanisms for colon cancer screening. We noted 
the following observations. 

Quality of Laboratory Services 

In the ColonCancerCheck program, the completed 
FOBT kits are analyzed at participating community 
laboratories for the presence of blood in the sam- 
ples. Test results are then sent to the participant's 
physician as well as to CCO. The physician informs 
the participant of his or her screening result. CCO 
also notifies the participant of his or her result. If 
a test result is positive, a colonoscopy is usually 
recommended as a follow-up test. 

Labs that want to participate in ColonCancer- 
Check must sign an agreement with the Ministry 
that outlines specific requirements, such as that 
the lab must be accredited by the Quality Manage- 
ment Program Laboratory Services of the Ontario 
Medical Association, and that it must conform to 
the Canadian External Quality Assurance Program 
established by the Ontario Association of Medical 
Laboratories (Association). Under an agreement 
between the Association and the Ministry, total 
Ministry-approved funding is up to $45 million 
over five years. A committee of the Association, 
comprising members from all labs that are partici- 
pating in ColonCancerCheck, monitors the quality 
of laboratory performance through monthly profi- 
ciency testing. 

Not only are there no Ministry or CCO represent- 
atives on this quality committee, but neither the 
Ministry nor CCO received reports on the quality 
assurance process and related results. The agree- 
ment between the Ministry and the Association 
states that the Ministry will only be informed of 
concerns that are not satisfactorily resolved by the 
appropriate laboratory, and must then be referred 
to the Ministry for action. The Association told us 
that there have been no incidents that needed to be 
referred to the Ministry since the quality assurance 
program began in 2008. 



Quality of Hospital Services 

CCO uses colonoscopy as a primary screening tool 
for people considered at increased risk of colon can- 
cer, and as a follow-up test to positive FOBTs. Only 
colonoscopies performed at participating hospitals 
are eligible for incentive funding from the colo- 
rectal cancer screening program. In 2011/12, these 
hospitals performed about 14,300 colonoscopies 
and received incentive funding of $4.6 million. Par- 
ticipating hospitals must meet specific quality stan- 
dards outlined by CCO, including the following: 

• endoscopists must perform at least 200 colon- 
oscopies annually to achieve or maintain 
competency; 

• the hospital's rate of bowel perforation must 
be no higher than one in 2,000 for screening 
procedures and one in 1,000 for all proced- 
ures; and 

• the examination of the bowel must meet a 
particular standard so that the thoroughness 
of the procedure can be assessed. 

We found that CCO collected the necessary data, 
but it did not analyze and monitor the data with 
respect to whether individual endoscopists com- 
plied with the requirements. Specifically, we found 
the following: 

• From colonoscopy data from 2008/09 to 
2010/11, we asked CCO to identify the 
percentage of endoscopists in the colorectal 
cancer screening program who did not do the 
minimum 200 colonoscopies annually. The 
review showed that more than 20% of endos- 
copists did not meet the requirement. 

• Our review of perforation data from 2009/10 
to the third quarter of 2011/12 for three 
regions we visited showed that the participat- 
ing hospitals generally met the perforation 
rate standards. However, the rates were 
determined based only on perforations that 
occurred on the procedure date. Our discus- 
sion with an expert in the field, as well as a 
review of research articles, indicated that 
complications may arise up to 14 days after 
the procedure. CCO said it did not track 
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this information beyond the procedure date 
because it was very challenging to do so. 
Other than the professional requirements of the 
specialists who conduct colonoscopy procedures, as 
well as the hospitals' requirements, there is no com- 
prehensive quality assurance process. CCO told us 
that it is considering developing a quality assurance 
process that is similar to that of the breast cancer 
screening program. 

Quality of Services at Private Clinics 

There are approximately 50 private clinics in 
Ontario that offer colonoscopy services but are not 
eligible to receive incentive funding for colonoscop- 
ies. Until 2010/11, these private clinics were not 
subject to any specific program quality standards 
and data collection requirements, as hospitals 
are under the colon cancer screening program. 
In 2010/11, however, the College of Physicians 
and Surgeons of Ontario began monitoring and 
inspecting these clinics. CCO is conducting a pilot 
project to explore how to incorporate these clinics 
into the screening program. 

Cervical Cancer Screening 

As mentioned earlier, CCO received Prescribed 
Registry status for the Ontario Cancer Screening 
Registry in May 2011. The Information and Privacy 
Commissioner approved CCO's information prac- 
tices and procedures in respect of the Registry in 
October 2011. Receipt of Prescribed Registry status 
enables the establishment of a comprehensive qual- 
ity assurance program. Prior to receiving this Pre- 
scribed Registry status, CCO was not able to obtain 
enough data to establish such a program. We made 
the following observations. 

Quality of Laboratory Services 

Cytology samples are generally obtained through 
Pap tests done in doctors' offices and are then 
analyzed by laboratories. CCO relies on the quality 
assurance processes that govern the accreditation 



and proficiency testing of labs by the Quality Man- 
agement Program — Laboratory Services (QMP-LS), 
operated by the Ontario Medical Association. In 
addition, the Ministry licenses laboratories to per- 
form a defined set of tests. 

CCO is responsible for quality assurance for the 
cervical cancer screening program. A quality assur- 
ance program would include test quality standards; 
the collection of data and monitoring of compliance 
to those standards; performance indicators; and the 
development of laboratory-related targets. However, 
CCO had not yet developed a quality assurance pro- 
gram for cervical cancer screening. CCO indicated 
that it did not have the authority to collect all of the 
data required for performance management and 
reporting by providers until it received Prescribed 
Registry status in October 2011. 

We also reviewed how quickly cytology test- 
ing should be completed. From 2007 to 2010, the 
provincial median turn-around time decreased to 
15 days from 21. However, turn-around time varied 
widely among laboratories. For instance, in the 
2010 fiscal year, the median turn-around time at 
individual labs ranged from seven to 33 days. 

Quality of Colposcopy Services 

Colposcopy is performed to investigate cervical 
abnormalities, such as pre-cancerous lesions. 
In 2010/11, the Ministry reported that 125,400 
colposcopy procedures were performed, a 28% 
increase from the 98,000 completed in 2004/05. 
Colposcopies are done in hospital-based clinics or 
in physicians' offices. 

In 2008, CCO established colposcopy guidelines, 
including guidelines on qualifications and train- 
ing for those who perform the procedure, and on 
quality assurance measures. However, CCO has not 
assessed and monitored the quality of colposcopy 
services to make sure they are provided in accord- 
ance with these guidelines. For instance, we found 
the following: 

• According to the guidelines, those performing 
colposcopy should complete approximately 
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100 colposcopies per year to maintain their 
competency level, with at least 25% of cases 
being new patients. As discussed earlier, CCO 
did not have the authority to collect all of the 
data required for performance management 
and reporting by providers until it received 
Prescribed Registry status in October 2011. 
• The guidelines call for colposcopy clinics to 
undergo annual reviews for quality assurance, 
and for clinical audits to be done at regional 
and provincial levels to ensure consistent 
results and provide appropriate feedback to 
clinicians. CCO has not implemented a qual- 
ity assurance program or conducted clinical 
audits since the program began in 2000. 
After CCO was granted authority to collect 
and review the data to assess quality of services 
in October 2011, it also signed a data-sharing 
agreement with the Ministry in January 2012 to 
access the necessary health information. With access 
to the Ministry's claims payment data, CCO will 
now be able to gather information on the number of 
colposcopies performed, whether a biopsy was done, 
the number of physicians performing colposcopies 
and the physicians' specialties. However, CCO still 
will not have data on the results of colposcopies and 
biopsies from all sources, including hospitals, clinics 
and other facilities. 



RECOMMENDATION 5 



To ensure that Ontarians are receiving quality 
cancer screening services, Cancer Care Ontario 
should work with the Ministry to: 

• establish monitoring procedures to ensure 
that quality assurance requirements are met 
for screening of breast, colorectal and cervical 
cancers, regardless of whether they are pro- 
vided under programs established by Cancer 
Care Ontario or other service providers; and 

• obtain screening data so it can review and 
assess the work performed by all service 
providers and measure the results against 
appropriate quality assurance standards. 




CCO strongly agrees that screening is most 
effective when offered through an organized 
program that incorporates all service providers 
and uses robust quality assurance mechan- 
isms to maximize the benefits of screening 
and minimize the harms. CCO also agrees that 
other jurisdictions, particularly the United 
Kingdom, offer excellent models for quality 
assurance programs. Building on models such 
as these, CCO will establish regular monitoring 
procedures to assess performance against qual- 
ity assurance requirements, such as by tracking 
screening frequency, cancer detection rates and 
competency of providers. CCO will work with 
the Ministry to obtain the data and the man- 
date required to ensure that quality assurance 
requirements for screening programs are met 
by all service providers, regardless of whether 
screening services are provided under programs 
established by CCO. This will include primary- 
care providers, radiologists, colonoscopists, col- 
poscopists, colonoscopy sites, mammography 
sites and laboratories. 

| MINISTRY RESPONSE 

The Ministry will continue to work with and pro- 
vide CCO with the appropriate resources, man- 
date and support to carry out effective quality 
assurance and monitoring of service providers. 

Under the Laboratory and Specimen Collec- 
tion Centre Licensing Act, all medical laborator- 
ies must, as a condition of licensing, meet 
the requirements of the quality management 
program carried out by the Ontario Medical 
Association (Quality Management Program — 
Laboratory Services, or QMP-LS). The quality 
management program carried out by the QMP- 
LS is for all laboratory testing and includes 
colorectal and cervical cancer screening tests. 
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PERFORMANCE MEASURES 
AND REPORTING 

Public Reporting of Performance Indicators 

CCO publishes annually its program indicators for 
its three screening programs through the Cancer 
Quality Council of Ontario's Cancer System Quality 
Index, a web-based public reporting tool that tracks 
the quality and consistency of all key cancer servi- 
ces in the province, from prevention and screening 
through to end-of-life care. The Council makes rec- 
ommendations for improvements to cancer services 
to the Ministry, via CCO's board of directors. In May 
2012, the Council said CCO needed to continue its 
efforts to improve the participation in its screening 
programs. For instance, for 2008 to 2010, only 27% 
of eligible women completed all the cancer screen- 
ing tests recommended for their age. 

CCO has also conducted formal evaluations of 
the three cancer screening programs and issued 



public reports on its assessments. The Ontario 
Cervical Screening Program Report, issued in 2011, 
covered 2003 to 2008; Ontario Breast Screening 
Program, 20th Anniversary Report, issued in 2010, 
covered 1990 to 2010; and ColonCancerCheck 2008 
Program Report was issued in 2010. 

The three cancer screening programs adopted 
some of the performance indicators that were 
developed by a national body, either the Public 
Health Agency of Canada or Canadian Partner- 
ship Against Cancer. The indicators used for the 
breast cancer screening program were in line with 
these key national indicators. However, both the 
colorectal and cervical cancer screening programs 
lacked indicators to assess the programs' follow-up 
and detection activities and outcomes. Under the 
Integrated Cancer Screening strategy, CCO is work- 
ing with the Ministry to set up and report on 13 key 
performance measures for all three cancer screen- 
ing programs. 



Chapter 3 
Section 




Ministry of the Attorney General 



Criminal Prosecutions 



Background 




The Criminal Law Division (Division) of the Min- 
istry of the Attorney General (Ministry) prosecutes 
criminal charges on behalf of the Crown before 
provincial courts in Ontario. It consists of Crown 
Attorneys, Deputy Crown Attorneys and Assistant 
Crown Attorneys, who are appointed under the 
Crown Attorneys Act (Act) and Crown Counsel, who 
are appointed under the Ministry of the Attorney 
General Act (collectively referred to as Crown 
attorneys or prosecutors) . The Act outlines the dut- 
ies of Crown attorneys in prosecuting charges laid 
by police forces, such as summoning witnesses to 
attend court, providing disclosure to defence attor- 
neys, presenting evidence in court and dealing with 
bail applications of accused offenders. 

It is the role of the Ontario Provincial Police and 
municipal police forces to lay criminal charges, 
under the federal Criminal Code of Canada and 
the Youth CriminalJustice Act and other provincial 
statutes, for such crimes as assault, uttering threats, 
impaired driving, robbery and homicide. Certain 
charges fall under the jurisdiction of federal pros- 
ecutors, and Ontario Crown attorneys generally do 
not deal with crimes involving drugs, terrorism, tax 
law and money laundering. 



The Division's Crown attorneys also represent 
the Crown in criminal appeals; provide legal advice 
to the police, the Attorney General and other law 
enforcement officials; provide special services such 
as applications to a court for electronic-surveillance 
authorizations, extraditions and search warrants; 
and develop criminal law policy recommenda- 
tions for both provincial and federal applications. 
In addition, the Division participates with other 
stakeholders in major initiatives targeting criminal 
activity related to guns and gangs, as well as the 
Ministry's Justice on Target initiative to reduce the 
average number of court appearances and days 
needed to dispose of a criminal charge. 

The Division receives about 600,000 new crim- 
inal charges each year from more than 60 police 
forces in Ontario. A Crown attorney is to prosecute 
a criminal charge only if there is a reasonable pros- 
pect of conviction and if it is in the public interest 
to prosecute. If at any stage of the case changed 
circumstances make the prospect of conviction 
no longer reasonable, the Crown attorney is duty 
bound to discontinue the prosecution. 

Criminal charges are prosecuted in either the 
Ontario Court of Justice or the Superior Court 
of Justice. The vast majority of charges are dealt 
with by the Ontario Court of Justice, which typ- 
ically tries less serious offences presided over by a 
judge alone; trials for more serious Criminal Code 
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offences take place in the Superior Court of Justice 
and are heard either by a judge alone or by a judge 
and jury. 

The Division operates from its head office in 
Toronto, six regional offices and 54 Crown attorney 
offices across the province. Operating expenses 
totalled $256 million in the 2011/2012 fiscal year, 
84% of which was spent on staffing. The Division 
employs approximately 1,500 staff, including about 
950 Crown attorneys and 550 support and admin- 
istrative staff. In addition, the Division spends 
approximately $3.2 million annually on contract 
lawyers who work on a part-time per diem basis. 



Audit Objective and Scope 



Our audit objective was to assess whether the 
Division had established adequate policies, sys- 
tems and procedures for the timely and efficient 
prosecution of criminal matters on behalf of 
the Crown, and for measuring and reporting on 
program effectiveness. 

Senior management reviewed and agreed to our 
audit objective and associated audit criteria. 

We conducted our fieldwork at the Division's 
head office in Toronto and visited five of the six 
regional offices and 11 of the 54 Crown attorney 
offices. Our work included interviewing staff, 
including prosecutors assigned to the Guns and 
Gangs initiative; reviewing recent reports and stud- 
ies; and examining policies, records, case files and 
systems. We also met with staff from the Ministry's 
Justice on Target initiative. 

We also held interviews with representatives of 
five police forces in the province, Legal Aid Ontario, 
the Criminal Lawyers' Association, and the Ontario 
Victim Services Secretariat to discuss their per- 
spectives on prosecutions and the criminal justice 
system in Ontario. 

We researched criminal prosecution programs 
in other Canadian and foreign jurisdictions and 
met with senior management of the federal and 



three other provinces' prosecutorial services. We 
also engaged an independent expert who has senior 
management experience in delivering criminal 
prosecution programs. 

We considered recommendations we made in 
our previous audits of the Ministry, including Legal 
Aid Ontario (2011), the Court Services Division 
(2008), and our last audit of the Division (1993). 
We also considered several major public reviews of 
the criminal justice system over the last decade. 

The Division's internal auditor conducted 
several reviews that were helpful in our audit, 
including those about travel card use and employee 
expenses; controls over the administration of 
proceeds of crime; and the Division's project to 
implement a new electronic Crown Management 
Information System (CMIS). 



Summary 



The number of Crown attorneys and the overall 
staffing costs for the Criminal Law Division (Div- 
ision) have more than doubled since our last audit 
in 1993. Yet the number of criminal charges that 
Crown attorneys dispose of per year has not sub- 
stantially changed — 572,000 in 1992, compared to 
576,000 in 2011. Partly as a result of the Charter 
of Rights and Freedoms, many cases are more 
complex than they used to be, so that more time 
and more court appearances are needed to pros- 
ecute them. Also, additional Crown attorneys have 
been assigned to deal with certain crimes, such as 
those involving gangs and other dangerous and 
high-risk offenders. 

However, it is difficult to gauge the actual 
impact of this on prosecutor workload, especially 
because the Division makes little use of numerical 
and statistical information to analyze the relative 
workload, efficiency and effectiveness of its Crown 
attorneys, and relies more on informal oversight 
by senior staff at each of the 54 Crown attorney 
offices. When we last audited the Division in 1993, 
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we noted "a systemic emphasis on prosecutorial 
discretion," and that monitoring was done by 
"more subjective means, such as informal feedback 
and personal knowledge about the individuals 
involved." This observation remains valid today. 

We continue to believe the Division would bene- 
fit from having information systems that would 
provide it with reliable summary data on prosecu- 
tor workloads, the outcome of prosecutions, the 
average time taken to resolve charges and other key 
performance indicators, both at a local office and 
an individual Crown attorney level. The Division 
can also make better use of the information on 
court activities currently available from within the 
Ministry until it completes the development of its 
own information systems. 

Our other major observations were as follows: 

• The Division does not formally assess its pros- 
ecutorial performance — for example, it does 
not gather information on how efficiently 
charges are screened; how long it takes to 
prepare cases; whether court diversion pro- 
grams for resolving minor criminal charges 
are used appropriately; the number of bail 
release applications, and what their condi- 
tions and results are; and what the outcomes 
of cases are. Furthermore, the rates at which 
certain Crown attorney offices went to trial 
were up to 20 times higher than the rates of 
other offices, significantly increasing justice 
system costs. We noted that Statistics Canada 
reported that Ontario had Canada's highest 
rate of adult criminal charges withdrawn or 
stayed (suspended by a court) in 2010/11 
(43% for Ontario versus 26% for the rest of 
Canada) and the lowest rate of guilty verdicts 
(56% for Ontario versus 69% for the rest of 
Canada) — but the Division does not have the 
information needed to determine the reasons 
for this or whether this relates more to certain 
regions or Crown attorney offices. 

• No staffing model has been established 
to determine how many Crown attorneys 
should be at each local office, and there is no 



benchmark for what a reasonable workload 
for each Crown attorney should be. Work- 
loads per Crown attorney varied significantly 
among local offices and between regions. 
For example, at two similarly sized Crown 
attorney offices, the average workload during 
the 12-month period ending March 31, 2012, 
was 572 charges per attorney at one and 1,726 
charges per attorney at the other. The Division 
does no periodic analysis to assess: 

• the reasons for the significant decrease over 
the last two decades in the average number 
of charges a Crown attorney disposes of per 
year; or 

• whether Crown attorneys need to be 
reassigned among Crown attorney offices 
to balance workloads and ensure similar 
charges can be handled consistently regard- 
less of where in Ontario they are laid. 

• Of the Division's six regions, the Toronto 
Region disposed of the most charges in 
total, but it did so at the highest cost per 
charge — $437, compared to the average of the 
other regions of $268. The Toronto Region 
also disposed of an average of about 40% 
fewer charges per Crown attorney than the 
average of other regions. We also noted that 
the use of court diversion programs for per- 
sons accused of minor criminal charges varied 
widely between Crown attorney offices — for 
example, one office reported that it resolved 
11% of its eligible charges using diversion pro- 
grams while a similarly sized office resolved 
75%. Reasons for these significant differences 
had not been analyzed. 

• The Division does not have a systematic 
process in place to ensure that services at its 
54 Crown attorney offices are consistently 
meeting minimum professional and Division 
standards. In our review of case files at 11 
Crown attorney offices we noted no standards 
for recording decisions and events, forms 
were either missing or not used, and case files 
were missing. 
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• A much-needed electronic case management 
system originally projected to cost $7.9 million 
and to be completed by March 2010 has been 
significantly delayed because of weak man- 
agement, oversight and financial reporting, 
and insufficient resources being dedicated to 
the project. Other provinces, such as Mani- 
toba, already have such systems in place and 
we noted that, rather than develop a new 
system, Alberta recently paid $1 for the rights 
to use and further develop Manitoba's system. 

• Because the Division does not measure its 
performance, the Ministry makes no mention 
of the Division in its annual reporting. In this 
respect, the Ministry differs from some other 
jurisdictions, which do measure and report on 
their criminal prosecution operations. 

We did note that the Division has contributed to 
some recent progress in improving court efficiency, 
as reported by the Justice on Target initiative, 
including the reversal of a decades-long trend of an 
increasing number of appearances and days needed 
to complete a criminal case in court. 



1 OVERALL MINISTRY RESPON 




The Criminal Law Division is committed to 
continuing to provide the citizens of Ontario 
with the highest-quality, efficient and effective 
prosecution services in support of public safety. 
The Division finds great value in and is actively 
incorporating the observations and recom- 
mendations of the Auditor General's review as 
it continues to improve how it delivers prosecu- 
tion services in these changing times. 

The audit correctly reports that cases are 
more complex than 20 years ago. In addition 
to the Charter of Rights and Freedoms, other 
factors add to the complexity of cases, resulting 
in an increased demand on Crown attorneys' 
time and hence an increase in the number of 
Crown attorneys. For instance, the introduc- 
tion of mandatory minimum sentences has 



resulted in significant reductions in guilty 
pleas and significant changes in their timing, 
which translate to more time being spent on a 
file. A typical impaired-driving case illustrates 
the point. Today one case takes two days in 
court, whereas in 1992, two to four cases were 
prosecuted in one day. In part because penalties 
have increased, such charges are vigorously 
defended and defence applications to exclude 
key evidence that the defence alleges the police 
obtained in violation of the accused's rights 
under the Canadian Charter of Rights and Free- 
doms are common. Nevertheless, the Division 
agrees with the Auditor General that it needs 
to find a way of measuring the impact that 
increased complexity has on workload. 

We agree that information collection and 
analysis are essential decision-making tools. We 
recognize that a multi-faceted electronic case 
management system is key, not only to informa- 
tion collection and analysis, but also to moving 
some of the paper-based manual processes into 
an electronic approach. We are disappointed 
with the progress made to date on our project 
to implement such a system. Nevertheless, our 
commitment to implement such a system, or 
group of systems, remains steadfast, and we are 
taking steps to get back on track. 

In the meantime, the Division is taking 
action to ensure the appropriate and necessary 
measurement of our workforce and workload 
through information already available to us. The 
Division will identify the gaps in meaningful 
data collection and will research similar metrics 
and systems that are being used to measure 
resourcing in the other jurisdictions referred to 
in the report. 

Through the actions that we are taking to 
increase our effectiveness and to continuously 
improve, we will deliver on the commitment to 
public safety for the people of Ontario. 
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Detailed Audit Observations 



MANAGING OPERATIONS 

Since our last audit in 1993, the number 
of criminal charges disposed of is virtually 
unchanged— 572,000 in 1992 compared to 576,000 
in 2011. Yet the number of Crown attorneys and 
support staff, as well as staffing costs — even after 
taking inflation into account — have doubled over 
the same period, as shown in Figure 1. In contrast 
to some other provinces we visited, management 
oversight tends to be more informal in nature, so 
as not to be perceived to affect the independence 
of each Crown attorney. As well, less use is made 



of information as a means to gauge the relative 
workload and effectiveness of the Division's Crown 
attorneys. Specifically, the Division lacks informa- 
tion systems at many levels to provide information 
to allow it to assess workloads and effectiveness. 

We understand that there are reasons why dis- 
posing of criminal charges consumes more resources 
now than it did in the past. For example, additional 
Crown attorneys have been assigned to focus on 
domestic and sexual violence, guns and gangs, 
dangerous and long-term offenders, and Internet 
child exploitation. Changes in legislation and case 
law that have occurred since 1982 as a result of the 
Charter of Rights and Freedoms have increased 
the complexity of many cases. The average time 
and number of appearances required to dispose of 



Figure 1: Annual Percentage Change in Number of Crown Attorneys, Charges Disposed and Criminal Law Division 
Expenditures, 1992-2011 

Source of data: Ministry of the Attorney General, Public Accounts 
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' Annual percentage change for expenditures were based on amounts adjusted for inflation to 2011 levels using Statistics Canada's Consumer Price Index. 
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a criminal charge have both doubled since our last 
audit, and these increases have been steadily climb- 
ing over the last 20 years except for the most recent 
year. Mandatory minimum sentences, disclosure 
requirements, and more complex evidence, such as 
DNA evidence, cell phone activities and computer 
forensics, have also contributed to an increase in the 
demand on Crown attorneys' time. 

Nevertheless, given a doubling of staff relative 
to essentially no caseload increase, it is all the 
more critical to objectively assess the efficiency 
and effectiveness of the Division's resource 
management. This requires timely, relevant and 
accurate information and analysis. Specifically, 
the Division needs information on how efficiently 
charges are screened; how long it takes to prepare 
cases; whether court diversion programs are used 
appropriately for minor charges; the number of bail 
release applications, and what their conditions and 
results are; and what the outcomes of cases are. 
This information can help management assign pros- 
ecutors to local Crown attorney offices to balance 
workloads across the province and monitor trends 
in charge resolution to identify situations that cause 
inefficiencies and delays. However, the Division has 
neither a manual nor a computerized system for 
collecting the needed information. It also has not 
established benchmarks against which it can assess 
aspects of its performance. 

Information that is available to the Division has 
come primarily from outside sources, most notably 
the Ministry's Integrated Court Offences Network 
(ICON), which reports on charges processed by the 
Ontario Court of Justice for each courthouse. Most 
Crown attorney offices serve only one courthouse, 
making the ICON reports useful for assessing cer- 
tain aspects of each office's performance. However, 
even though this information is distributed monthly 
to regional and local Crown attorney management, 
we found no indication that the Division routinely 
used the information to analyze the performance 
of its regional and Crown attorney offices and 
individual Crown attorneys. As well, the usefulness 



of this information to the Division is limited for the 
following reasons: 

• ICON reports information on a criminal 
charge basis rather than by case or person. 
Since most cases involve multiple charges 
and not all the charges proceed, manage- 
ment has not been able to determine whether 
whole cases have been lost, or whether minor 
charges were simply withdrawn but the case 
proceeded on more serious charges. 

• Although ICON reporting does separate the 
number of charges that are stayed by the 
court (proceedings against an accused are 
stopped before an acquittal or conviction) 
from the number of charges withdrawn by 
prosecutors, it does not identify the reasons 
for either occurrence; knowing this could help 
the Division reduce their frequency. Stays and 
withdrawals can occur for many reasons. For 
example, the accused may have been denied 
complete disclosure, a right to counsel or a 
timely trial; evidence may have been deemed 
inadmissible; and witnesses may have refused 
to testify. In some of these cases, preventing 
the stay and withdrawal is in the control of 
the prosecutor. 

Each Crown attorney office's management 
responsibilities are assigned to a senior Crown 
attorney, who we noted also continues to carry 
his or her own caseload. Senior Crown attorneys 
told us that they rely extensively on feedback from 
prosecutorial staff about their own workloads. 
They also rely on anecdotal information, such as 
comments from the judiciary, defence attorneys 
and court staff, to make them aware of concerns 
with prosecutors' performance. The management 
framework tries to strike a balance between pros- 
ecutors having independence in the day-to-day 
decision-making on cases assigned to them and the 
need for Division management to hold prosecutors 
accountable for efficient and effective prosecutions 
that meet expectations and standards. 

We noted that local Crown attorney offices have 
developed their own management and operational 
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practices. In some cases, this is because local police 
forces or courts are administered slightly differently 
from one area to another or because the 54 local 
Crown attorney offices vary greatly in size — the 
smallest have only one or two Crown attorneys, 
while the downtown Toronto office has more than 
125. In smaller Crown attorney offices, one person 
might handle all incoming cases from beginning 
to end, whereas in larger offices it is common for 
several prosecutors to work on a case at different 
stages and make court appearances. However, there 
is no overriding provincial management model, and 
head office had not done a formal analysis on the 
variation in practices among offices that could be 
used to identify best practices in the various Crown 
attorney offices where it might be beneficial to 
standardize practices to reduce costs. For example, 
we found no consistency in how case files were 
handled by Crown attorneys, including ownership 
and custody assigned to files between police forces 
and Crown attorney offices, standards for docu- 
mentation on case files, and notation standards for 
key decisions, such as screening charges or bail and 
sentencing recommendations. 



RECOMMENDATION 1 



To ensure that decisions on the use of legal 
and support staff resources and results of 
prosecutions are supported by timely, relevant 
and accurate information, the Criminal Law 
Division of the Ministry of the Attorney General 
should identify what information is needed and 
develop systems as soon as possible to deliver 
this information to its regional and local Crown- 
attorney-office management. The Ministry 
should also use this information to hold the Div- 
ision accountable for demonstrating the cost- 
effective use of its resources. Until such time as 
the Division can gather its own information on 
its activities, it should make better use of the 
available ministry information on courthouse 
activities to more effectively oversee operations 
and report on its use of resources. 




The Criminal Law Division recognizes the 
importance of having reliable management 
information system (s) to make informed deci- 
sions in support of the effective leadership of 
its operations. 

The Division is reviewing existing systems 
and the information available within the Min- 
istry that relates to its work with a view to iden- 
tifying the gaps in current information analysis, 
reporting and report usage. The Division's 
longer-term objective is to ensure that future 
information systems accurately capture and 
support the analysis required to enable meas- 
urement of the cost-effectiveness and optimal 
use of its resources, including measurement and 
assessment of workload. 



OVERSIGHT OF PROSECUTORS 

Management of cases and their timely progression 
through the justice system has been a particularly 
crucial issue since October of 1990, when the 
Supreme Court of Canada released its ruling in the 
case of R. v. Askov. In general, at the time, the Askov 
ruling and related rulings established that the 
acceptable time to trial was generally eight to 10 
months. As a result, thousands of backlogged char- 
ges across the country were dismissed on grounds 
of unreasonable delay in the prosecutions. Since 
then, there have been further court rulings that 
have narrowed the circumstances under which a 
judge can dismiss a charge on the basis of unreason- 
able delay caused by prosecutors. The Ministry has 
been examining its processes to resolve charges at 
the earliest opportunity to reduce both costs and 
the risk of delays. Its Justice on Target initiative is 
one of these strategies. 

Justice on Target aimed to achieve, over a four- 
year period ending in June 2012, a 30% reduction 
in the number of court appearances and days to 
dispose of a charge. Justice on Target reported that 
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as of March 31, 2012, it had achieved success in 
reversing a decades-long trend of increases, but it 
was not close to meeting these reduction targets. 
Court appearances and the number of days to 
dispose of a charge had decreased by 7% and 2%, 
respectively The Division is a key stakeholder in the 
initiative, contributing to the reductions achieved 
by helping to implement opportunities for charges 
to be resolved earlier. 

The management staff at Crown attorney offices 
told us that they informally monitor charges that 
reach the eight-month, 10-month and 12-month 
marks out of concern that charges will be stayed 
due to the delay. As Figure 2 indicates, the backlog 
of charges in the courts taking longer than eight, 10 
or 12 months still exists and has not changed sig- 
nificantly, although there has been some improve- 
ment in the most recent fiscal year. 

The Division does not track whether specific 
actions taken by Crown attorneys have any effect 
on the progress of a case. For instance, Crown 
attorneys we interviewed told us that prosecutors 
initiate only a small percentage of court adjourn- 
ments — one of the leading causes of delays — but 
there is little data to support this given that the 
Division does not track the delays caused by 
adjournments and the reasons for them. As well, 
information in ICON on the causes of adjourn- 

Figure 2: Ontario Court of Justice Percentage of 
Criminal Charges Pending, 2007/08-2011/12 

Source of data: Ministry of the Attorney General 
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ments was either incomplete or not used by 
the Division. 

We were surprised that the Division does not 
formally track the number of motions made by 
defence attorneys to the court requesting that 
charges be stayed due to delay, or the reasons for 
the successful motions. Although the ICON system 
identifies pending charges and the extent of the 
delays, there is not enough data in ICON to analyze 
what types of charges make up the backlog and the 
reasons for the delays. As a result, it is up to the 
Division to gather information on the reasons for 
stays. Another example of an area where inconsis- 
tencies should be probed by the Division to under- 
stand and reduce delays is the setting of trial dates. 
While courts control the progression of cases, we 
noted at several Crown attorney offices we visited 
that some courts had rules for setting trial dates 
within, say, 90 days, while other courts had no 
such rules. Prosecutors also have a significant role 
to play in ensuring that cases progress through the 
court in a timely manner by bringing any unneces- 
sary delays to the attention of the court for action. 
A senior Crown attorney also has the opportunity 
to bring up more systematic causes for delays at a 
particular courthouse at regular meetings that are 
held with the judiciary on the administration of 
the courthouse. 

Statistics Canada reports that in 2010/11, 
Ontario had the lowest proportion of guilty ver- 
dicts in adult criminal cases among all Canadian 
jurisdictions, at 56%. The national figure, exclud- 
ing Ontario, was 69%. The Division has not ana- 
lyzed its prosecution results to determine why this 
might be the case. 

The Division has said it is aiming to reduce the 
number of trials that collapse — usually through 
last-minute guilty pleas or the withdrawal of 
charges — on the day the trial is set to begin. Trials 
that collapse on the first scheduled court date incur 
costs that could have been avoided because they 
unnecessarily tie up courtrooms, court staff, the 
judiciary, witnesses and police, all arranged months 
in advance. Trial collapse rates vary widely among 
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Crown attorney offices, from 4% to 22%. The 
Division obtains trial collapse rates from ICON, but 
ICON does not provide enough detail to allow the 
causes of the collapses to be analyzed or compared. 
For instance, ICON captures only collapsed charges, 
not collapsed cases. Minor charges are regularly 
withdrawn at trial, particularly when multiple 
charges have been laid, so statistics on collapsed 
charges do not isolate the real problem, which is 
the collapse of entire cases. 

In addition, the Division did not analyze why cer- 
tain regions and Crown attorney offices had higher 
trial rates — local offices ranged from about 1% to 
20% of total annual charges, and the region with the 
highest rate was Toronto. The cost implications of 
a Crown attorney office proceeding to trial at a rate 
of 20 times more than another office warrants more 
formal attention, particularly since the Toronto 
Region has the largest total caseload in the province. 

Disclosure and Screening of Charges 

After police lay charges, they provide a report 
on those charges to the Crown attorney's office 
in what is known as a Crown brief. Police forces 
want to move to an electronic disclosure system to 
improve efficiency, but the Division has been slow 
to implement its system for accepting Crown briefs 
electronically. At the time of our audit, only five of 
54 Crown attorney offices were accepting electronic 
disclosure from police forces. 

After receiving a Crown brief from police, Crown 
attorneys assess, or screen, the charge (s) to deter- 
mine whether to prosecute — essentially, whether 
there is a reasonable prospect of conviction and, if 
so, whether the prosecution is in the public interest. 
During this screening, Crown attorneys can decide 
not to prosecute charges by withdrawing some or 
all of the charges or otherwise resolving charges 
without going to trial by recommending to the 
court the use of diversion programs or alternative 
sentencing options. Alternatively, additional char- 
ges could be laid by the police on recommendation 
of the Crown attorney. 



A stay of proceedings occurs when charges 
are temporarily or permanently suspended by 
the court, such as when the rights of an accused 
person have been violated or the Crown attorney 
requests that the accused person participate in a 
diversion program; in contrast, withdrawn charges 
are initiated by the Crown attorney when there 
are no reasonable prospects of conviction or it is 
not in the public interest to prosecute. Statistics 
Canada, which receives data from the Ministry's 
Court Services Division, reports that 43% of adult 
criminal cases in Ontario in 2010/11 were resolved 
by staying or withdrawing charges laid, the highest 
proportion of such cases in Canada. The average 
of other provincial and territorial jurisdictions in 
Canada was 26%. There could be many reasons for 
withdrawing charges, including the quality of the 
Crown briefs that police send to Crown attorneys 
for charge screening, plea negotiations, witnesses 
not co-operating and the success of diversion 
programs. Such a significant difference could 
also indicate that Ontario as a whole or certain 
Crown attorney offices are incurring unnecessary 
costs because weaker charges are not being suf- 
ficiently screened out before court proceedings 
begin. However, the Division does not collect data 
so that charge withdrawal rates can be analyzed 
to determine if there are any systemic issues that 
warrant attention. 

Diversion Programs 

The Ministry's voluntary diversion program, called 
the Direct Accountability Program, is an alterna- 
tive to formal prosecution for people who have 
been charged with minor criminal offences. The 
program benefits the accused, the courts and the 
community. The program involves accused people 
being held accountable through community-based 
sanctions such as restitution, community service 
work, charitable donations or participation in pro- 
gramming such as anger management or alcohol 
and drug awareness. The charge (s) against the 
accused can be withdrawn or stayed if the person 
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successfully completes the sanctions imposed. 
Diversion strategies — used for charges that Crown 
attorneys deem eligible, such as theft-of-property 
offences — are less costly than formal court pro- 
ceedings and they allow for quicker resolution of 
charges, which frees up court resources. 

Diversion programs were in place in almost 
all Crown attorney offices. However, as Figure 3 
illustrates, referral rates by Crown attorneys as 
reported by the Justice on Target initiative differed 
widely among the six regions. For example, the 
Toronto Region had a referral rate of 57% and the 
Eastern Region 35%. The referral rates varied even 
more significantly among Crown attorney offices. 
For example, in the 2011/12 fiscal year, two Crown 
attorney offices in the same region that are roughly 
the same size had referral rates of 11% versus 75%. 
The Division's process for tracking and analyzing its 
use of the programs and the results of the referrals 
did not sufficiently address the varying referral 
rates or improve their consistency. As a result, the 
Ministry cannot determine whether the diversion 
programs are being appropriately used to the 
extent possible. In fact, the varying use of diversion 
programs by Crown attorney offices and Crown 
attorneys may result in an inconsistent approach 

Figure 3: Referral Rates to the Direct Accountability 
Program (Diversion) by Region, 
May 1, 2011-April 30, 2012 (%) 

Source of data: Ministry of the Attorney General 
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across the province for dealing with minor offences 
and the resulting criminal records of those who are 
not diverted. 



RECOMMENDATION 2 



In order for the Criminal Law Division to 
adequately oversee its prosecutions, monitor 
its costs and assess its performance, it should 
regularly analyze the trends, rates and reasons 
for stays and withdrawals, adjournments, trial 
rates, bail release violations, guilty pleas and 
guilty verdicts, and use of diversion programs. 
In addition, the Division should compare its 
performance to other provinces and, where 
Ontario's overall trends differ from those of 
other large provinces, determine the reasons for 
such differences. 



The Criminal Law Division recognizes the 
importance of using information to evaluate the 
quality and results of its prosecution service. 

Analyzing and seeking to understand the 
trends, rates and reasons for outcomes in crim- 
inal proceedings will be helpful in overseeing 
our prosecutions. Further analysis and research 
into how this will assist with the monitoring 
of costs and performance of the Crown in con- 
ducting prosecutions is required. The Division 
recognizes the need for consistent practices and 
approaches within its regions and local offices, 
taking into consideration the variables that 
exist, such as size of office, geographical/demo- 
graphic information, size of police force and 
charge volume. There are lessons to be learned 
from each office, with perhaps the most valu- 
able coming from the comparison of like offices. 

The Division will also benefit from the experi- 
ence of other large prosecution services across 
Canada and will seek opportunities to make 
meaningful multi-jurisdictional comparisons. 
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MANAGING WORKLOADS 

As we noted earlier, the number of Crown attorneys 
has doubled since 1992 despite the fact that the 
number of criminal charges has remained essen- 
tially constant. The Division is unable to assess 
whether the increases in the number of Crown 
attorneys is reasonable because it does not gather 
the necessary information to analyze whether the 
complexity of the cases or the time needed to dis- 
pose of the charges has increased to such an extent 
as to require twice as many staff. The Division does 
not use a staffing model to determine the appropri- 
ate number of Crown attorneys, and there is no 
benchmark for what a reasonable workload for 
each Crown attorney should be. 

Using data from the Ministry's ICON system, we 
calculated the average number of charges disposed 
of per Crown attorney and found a wide variance 
among Crown attorney offices. For example, at half 
of the 54 Crown attorney offices, the rate of charges 
disposed varied by more than 25% of the overall 
average of 700 charges. Furthermore, the average 
workload per Crown attorney at two similarly sized 
Crown attorney offices differed significantly for 
the 12-month period ending March 31, 2012—572 
charges per attorney at one office versus 1,726 
charges per attorney at the other. There were also 
two regions that disposed of significantly fewer 
charges per Crown attorney, compared to the other 
four regions, as shown in Figure 4. 

One of the key reasons for ensuring that Crown 
attorney workloads are reasonably comparable is 
that they should be able to devote a similar amount 
of time to charges of a similar nature and complex- 
ity, regardless of where in Ontario the charge is laid. 

Crown attorney offices also had different ways 
of organizing staff and assigning cases to Crown 
attorneys. For example, one office we visited 
had case management co-ordinators to carry out 
administrative tasks for incoming charges, while 
in another office Crown attorneys performed 
these functions. 

Over the past few years, the Division has drawn 
up business plans requesting that additional Crown 



Figure 4: Average Number of Charges Disposed of per 
Crown Attorney by Region, 2011/12 

Source of data: Ministry of the Attorney General 
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attorneys be hired for initiatives such as efforts 
to address guns and gangs, high-risk offenders 
and domestic violence, but these plans have not 
included any expected outcomes or workload 
measures. As a result, it is difficult for the Division 
to demonstrate what payback has resulted from the 
incremental costs that were incurred or why the 
additional staff resources were necessary, especially 
since it has no measures in place to assess the work- 
load of its existing prosecutors. 

We noted that both Manitoba and the Public 
Prosecution Service of Canada (federal prosecu- 
tors) track each prosecutor's workload. In addition, 
British Columbia and Manitoba assign each case to 
a specific prosecutor, which allows their manage- 
ment to monitor and assess workloads; this is not 
the usual practice in Ontario. These other provinces 
and federal prosecutors also use electronic case- 
management systems to track workloads. 

In addition, we calculated the average cost the 
Division incurred to prosecute charges and noted 
significant differences between regions, as indicated 
in Figure 5. For instance, the Toronto Region had 
the highest average cost at $437 per charge, versus 
the average of other regions of $268 per charge. As 
Figure 4 already illustrated, the Toronto Region also 
disposed of an average of about 40% fewer charges 
per Crown attorney than the average of other 
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Figure 5: Average Cost per Charge Disposed of by 
Region, 2011/12 ($) 

Source of data: Ministry of the Attorney General 
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regions — 478 compared to 811. The Division had 
not done a formal analysis to determine the cause of 
these variances. 

Our discussions with the Division's senior man- 
agement also revealed that the Division's collective 
agreement with its prosecutors might be interpreted 
in such a way as to restrict the Division's ability to 
address workload pressures, particularly in its abil- 
ity to relocate prosecutors between Crown attorney 
offices. In addition, the collective agreement limits 
the Division's use of contract lawyers to address 
workload pressures to a maximum of 30 days in any 
quarter of a year. This could also account for manage- 
ment's reluctance to deal with some of the differ- 
ences in costs and workloads per office. We did note 
during our fieldwork that the Division had completed 
an assessment of its head office support services to 
Crown attorney offices and as a result had been able 
to reduce its head office staff by 11 positions. 



RECOMMENDATION 3 



To ensure that Crown attorneys have the work- 
load flexibility to devote a similar amount of 
time to charges of a similar nature, the Criminal 
Law Division should: 

• establish benchmarks for what a reasonable 
workload for each Crown attorney should be; 



collect and analyze information on work- 
loads and cost variances between regions 
and Crown attorney offices to identify 
opportunities to use resources as efficiently 
as possible and address inconsistencies; and 
ensure that management has the ability 
and flexibility to address temporary and 
permanent workload pressures by, for 
example, relocating prosecutors and sup- 
port staff between Crown attorney offices, 
and using contract lawyers where and 
when appropriate. 

MINISTRY RESPONSI 

The Criminal Law Division recognizes the need 
and the increasing operational requirement to 
develop measures of workload. 

Defining workload is challenging and must 
go beyond the number of cases per Crown attor- 
ney. There are a number of factors that impact 
workload. For instance, a significant murder 
trial could take two Crown attorneys three 
years, equating to a fraction of a case per year. 
Also, in a small office, two prosecutors might 
carry a workload of 1,500 charges per year and 
have to fulfill all obligations on these charges 
from screening and vetting right through to 
disposition. They also might need to travel three 
hours to deal with some of the charges. It is 
challenging to determine the relative workload 
weight of these two examples. 

The Division will research other jurisdictions' 
attempts to measure workload and will develop 
an approach that meets its needs. Using avail- 
able data, the Division will then review workload 
and develop processes for making comparisons. 
The Division will also use the information to 
analyze the differences between regions and 
local offices to determine how resources can 
be used as effectively as possible. This will also 
promote staff wellness in the Division by provid- 
ing additional information on which workload 
distribution decisions can be made. 
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QUALITY ASSURANCE 

The Division's management style provides regular 
feedback and support from senior Crown attorneys 
to prosecutors through ongoing collaboration. 
However, the Division does not periodically review 
a sample of the work done by each of its Crown 
attorneys, particularly its more than 750 assistant 
Crown attorneys, to assess whether they generally 
meet expectations and professional and divisional 
standards. We were advised that the criminal 
justice system itself acts as an external quality 
control measure, in that the work of prosecutors 
is reviewed by police, judges and defence counsel 
during each case they prosecute. However, senior 
Crown attorneys who oversee other prosecutors 
generally get information on the quality of the work 
done in court by their staff prosecutors only when 
they are told by court staff, or, periodically, when 
prosecutors observe colleagues' court proceedings 
when they happen to be in the same courtroom. 

Without such periodic spot checks, divisional 
management faces a challenge in its bid to ensure 
that high-quality prosecutorial services are con- 
sistently delivered across the province. Periodic 
reviews would help to identify whether case files 
are acted on in a timely manner, whether profes- 
sional standards are met, whether policies for 
bail and charge screening are complied with, and 
whether efficiency-increasing initiatives such 
as diversion programs are used appropriately 
and consistently. 

Crown attorneys have prosecutorial independ- 
ence in their decision-making, but they are account- 
able for carrying out the policy direction set out in 
the Divisional Crown Policy Manual. The manual 
is a compilation of prosecution policies, detailed 
legal advice, and practice memoranda and guide- 
lines intended to provide a consistent approach to 
prosecutions across the province. For instance, the 
manual addresses procedural policies on charge 
screening, providing disclosure, dealing with vic- 
tims and sentencing, as well as accepted practice for 
specific types of prosecutions, including Aboriginal 



justice, child abuse, impaired driving and matters 
under the Youth Criminal Justice Act. 

Our review of case files at the 11 Crown attorney 
offices we visited showed a number of inconsistent 
practices. For instance, ownership and custody 
of closed case files was handled inconsistently, a 
number of files we requested could not be located, 
and standardized forms, such as for charge screen- 
ing, were either missing or not used by the Crown 
attorney office. There were also no standards 
for recording decisions and events, which were 
therefore inconsistently recorded, both in different 
offices and within the same office. 

In addition, Crown attorney offices have their 
own management-oversight processes, supple- 
mented by employee performance evaluations done 
semi-annually by the local senior Crown attorney. 
However, these evaluations did not include a 
review of any recent files the prosecutor had com- 
pleted. Including a spot check of a sample of files 
as part of the employee evaluation process would 
also communicate the importance of documenting 
compliance with Division standards. 

We did note certain circumstances where case 
files were reviewed by senior Crown attorneys, 
but these processes were ad hoc and were not 
based on any comprehensive or formal checklist 
to assess compliance to professional and divisional 
standards. For instance, quality of case work will be 
assessed if an appeal is granted by a court, which 
occurs in less than one-quarter of 1% of all closed 
cases. In addition, we noted that the Division had 
initiated a good practice of periodically reviewing 
some high-profile cases processed by the Major 
Case Management and Guns and Gangs units of 
the Division to assess processes or practices that 
worked well and what could be improved. 

From our research, we learned that one other 
province does reviews of prosecutors' case files 
and that the federal prosecutors recently initiated 
a pilot project to do so. In addition, we noted that 
the Crown Prosecution Service of England and 
Wales (CPS) had a robust, publicly reported quality 
assurance program for assessing its work. The CPS's 
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program included assessing case work on closed 
files against the CPS's core quality standards and 
the code setting out what their prosecutors do, how 
they make decisions, and the level of service they 
commit to in key aspects of their work. In its most 
recent assessment of more than 10,000 closed case 
files, the CPS found that 78% of commitments in 
the case files were fully met, 11.5% partially met 
and 10.5% not met. 



RECOMMENDATION 4 



To ensure that regional and division manage- 
ment have adequate assurance that cases are 
prosecuted in a consistent, timely and effective 
manner that meets expected standards, the 
Criminal Law Division should perform a per- 
iodic, objective review of a sample of files from 
each Crown attorney relating to the prosecu- 
tions each one handled during the year. 



The Criminal Law Division will compile a list of 
practices to advance the leadership's commit- 
ment to reinvigorate the performance planning 
and feedback process, including more meaning- 
ful feedback that differentiates performance, 
attendance by the supervising Crown attorney 
during proceedings and spot-checking files. This 
information will supplement existing quality 
control practices with respect to periodic review 
of files. Consultations with the supervising 
Crown attorneys will occur prior to implementa- 
tion of items on the list. In doing so, the Division 
anticipates that it will be able to ensure the 
continued development of its workforce and to 
allow decision-makers to enhance consistency 
in how files are prosecuted on a local, regional 
and provincial level. Although these practices 
have existed in local offices for some time, the 
Division will benefit from a more consistent 
approach across all offices. 



CROWN MANAGEMENT 
INFORMATION SYSTEM 

Many Crown prosecution services — for example, 
Manitoba in 1999, British Columbia in 2002 and 
the Public Prosecution Service of Canada in 2002 — 
have moved to electronic systems that essentially 
manage and track cases and how resources are used 
to prosecute those cases. 

Ontario Crown attorney offices still rely largely 
on paper-based manual processes to manage 
their workload. As of the end of our fieldwork in 
July 2012, the Division's project to implement an 
electronic case management system, known as the 
Crown Management Information System (CMIS), 
was significantly delayed and projected to be 
over budget. 

The original business case for CMIS was submit- 
ted to Treasury Board for approval in December 
2006 and estimated that the project would cost 
$7.9 million and be completed by March 2010. 
CMIS would allow the Division's 54 Crown attorney 
offices to receive, track, store, modify and share 
electronic documents; and to automate several 
processes, including scheduling, criminal case 
management and business intelligence. In addition, 
the new system would allow the Division to receive 
electronic documents from police forces, including 
Crown briefs and pre-trial disclosure documents. 
The Ministry's Justice Technology Service unit was 
to work with the Division to provide the informa- 
tion technology expertise and manage the project. 

In 2010, the Division engaged a consultant to 
assess why the CMIS project had been delayed. 
The consultant recommended that the Division put 
together a dedicated project team, including special- 
ists in change management, communications and 
implementation management. It also recommended 
that the software application that was originally 
chosen for the project be replaced with more robust, 
vendor-supported, user-friendly software. The pro- 
ject completion date was extended to March 2012, 
and the Ministry established a new project team. 
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By September 2011, the new project team leader 
reported that the project was delayed again. In Nov- 
ember 2011, the Ministry developed a new tentative 
strategy under which an additional three years 
would be required to complete the project, making 
the projected completion date March 2015, at a 
revised projected cost of $11.5 million. As a result, 
it was decided in May 2012 that the project would 
be reviewed by the Ministry's internal auditors. 

In its June 2012 report, the Ministry's internal 
auditors raised a number of concerns with respect 
to the management of this project. The internal 
auditors also noted that IT consultants hired to 
develop CMIS, at a cost so far of $1.3 million, were 
not managed effectively and that billings were 
based on time spent rather than meeting project 
deliverables and outcomes. 

In addition to the internal auditor's findings on 
project delays, we questioned why a totally new 
project was initiated and developed without more 
carefully considering the systems already in use 
in other provinces and the cost and time savings 
that could be achieved by using an already proven 
system. We noted that, although the Ministry did 
research existing systems in 2005, it did not pursue, 
for example, working with Manitoba's system, 
which we observed has useful features for tracking 
cases and workloads, something that would address 
many of the concerns we have raised in this report 
regarding the Division's lacking information on its 
operations (although Manitoba's system does not 
electronically store case documents, which is a fea- 
ture the Division has specified for its new system). 
We were advised that Alberta's new case manage- 
ment system will be modelled on Manitoba's 
system, which was provided to Alberta at a cost of 
$1. Alberta plans to enhance the system to electron- 
ically store case documents. Manitoba's agreement 
with Alberta requires that Alberta provide Mani- 
toba with any modifications and enhancements to 
the case management system, which we consider 
to be a beneficial collaborative arrangement for 
both parties. 



As of July 2012, CMIS was still underway, and a 
total of approximately $5.2 million had been spent. 
The Ministry will need to seek Treasury Board's 
approval should it revise its budget and timeline to 
complete the project. 



RECOMMENDATION 5 



To ensure that the paper-intensive processes 
currently used by the Criminal Law Division are 
replaced with an electronic case-management 
system to better manage and track prosecutions 
and staff resources, the Ministry of the Attorney 
General should significantly strengthen project 
management to mitigate the challenges posed 
by its Crown Management Information Sys- 
tem (CMIS). In addition, the Ministry should 
formally evaluate existing case-management 
systems in other jurisdictions to identify any 
potential for achieving savings and shortening 
the time to get the required system in place. 

| MINISTRY RESPONSE 

The Criminal Law Division agrees with the 
Auditor's recommendation that having a robust 
information management system in place 
will enhance the Division's capacity to more 
effectively and efficiently prosecute cases and 
use resources. The Division has already taken 
steps to enhance its project management and 
oversight of the CMIS project. Rather than 
implement the tentative $11.5 million strategy, 
the Division is now evaluating existing case 
management systems from other jurisdictions, 
as well as systems used by police forces, and will 
continue to do so. Specifically, systems will be 
re-examined with a view to establishing specific 
components of a system rather than continue to 
seek one ideal system for Ontario's prosecution 
service. For example, the Division is currently 
re-examining Manitoba's case management 
system to determine if it could meet some of 
Ontario's needs. 
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PERFORMANCE MEASUREMENT AND 
PUBLIC REPORTING 

We found the Ministry does not, in any substantive 
way, measure or report on the performance of the 
Criminal Law Division. In fact, the Ministry makes 
no mention of the Division in its annual reporting. 

As we have noted throughout this report, the 
Division collects little data of its own. For example, 
the Division does not currently have performance 
indicators for Crown attorneys' workloads, charges 
disposed of per Crown attorney, cost per case, 
average cost of trial resolution, and prosecution 
outcomes such as conviction rates and use of court 
diversion programs. It also does not track certain 
key prosecutorial outcomes, such as trial collapse 
rates, rates of withdrawal of charges, the number 
of stays and adjournments, crimes committed 
by accused persons on bail release, and witness 
attendance rates. 

A number of other jurisdictions we researched, 
including British Columbia, Manitoba, the Public 
Prosecution Service of Canada, and the Crown 
Prosecution Service of England and Wales (CPS) did 
measure and report on their results. For instance, 
British Columbia's Prosecution Service publicly 
reports a strategic plan, which includes its priorities 
and major projects, and timelines for their imple- 
mentation. CPS also reports extensively on its busi- 
ness planning and performance management, and 
analyzes its efforts to meet strategic goals. It reports 
on criminal case outcomes for each of its districts, 
witness attendance rates and average costs per case. 

In its annual report, the Alberta justice ministry 
reports on the public perception of fairness in its 
prosecution service — that is, the percentage of 
Albertans who agree that the Alberta justice min- 
istry provides fair and impartial service in prosecut- 
ing people charged with crimes. 

Jurisdictions such as the United Kingdom and 
the United States report on prosecutors' success 
rates in obtaining convictions, but we acknowledge 
that this is not common practice in Canada. Div- 
ision staff and prosecutors in other Canadian juris- 



dictions told us that the idea of measuring "success" 
by computing and emphasizing conviction rates 
was not consistent with what has been established 
as the Crown attorney's role: to simply lay out the 
facts and let a judge and/or jury decide innocence 
or guilt. However, this view does not take into 
account the fact that Crown attorneys make deci- 
sions on whether or not to prosecute based on 
what they consider the likelihood of conviction. 
For instance, dramatically high conviction rates 
might indicate undue conservatism in proceeding 
to trial with charges, while low rates might indi- 
cate the need for increased senior level oversight 
and guidance. 

As noted earlier, Statistics Canada data shows 
that Ontario achieved the lowest proportion of 
guilty verdicts in adult criminal cases among 
Canadian jurisdictions in 2010/11, as well as the 
highest proportion of cases in which charges were 
stayed or withdrawn. Without further analysis, 
these statistics could mean that, overall, Ontario 
Crown attorneys are not as successful as prosecu- 
tors in other provinces. On the other hand, they 
could indicate that Ontario successfully re-directs 
a higher proportion of cases away from courts 
and to its diversion programs and achieves lower- 
cost solutions for cases involving relatively minor 
charges. Without analyzing the reasons for these 
variances, the Division and the Ministry cannot 
make an informed judgment on issues such as 
these. We were advised that the Ministry also does 
not compare key performance measures with other 
provinces because there have not been successful 
collaborative efforts among the various prosecution 
services to identify appropriate, consistent and 
meaningful performance measures. 



RECOMMENDATION 6 



Particularly given the importance of the Crim- 
inal Law Division to the mandate of the Ministry 
of the Attorney General, the Ministry should 
develop performance indicators specifically for 
the Division, and should publicly report on the 
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Division's progress toward those indicators. It 
should also consider liaising with other prov- 
inces' prosecution services to develop common 
performance measures that would allow for 
comparison, benchmarking and the identifica- 
tion of best practices. 



| MINISTRY RESPONSE 




Given that what gets measured gets done, 
the Criminal Law Division acknowledges the 
importance of continuing to evolve its key per- 



formance indicators and of measuring results in 
support of its core mandate of contributing to 
public safety for the citizens of Ontario. The Div- 
ision will continue to explore additional qualita- 
tive and quantitative performance indicators, 
both within the Ministry and with other pros- 
ecution services. The audit accurately outlines 
our concerns about defining success through the 
measurement of conviction rates. Once these 
indicators are identified, the Division will move 
forward to publicly report on the results relating 
to these performance indicators. 



Chapter 3 
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Background 




Chronic diseases are the leading cause of death 
and disability in Ontario. One of the most common 
chronic diseases is diabetes, which results from the 
body's partial or complete inability to produce and/ 
or properly use insulin, a hormone that regulates 
blood sugar. Diabetes can lead to kidney failure, 
heart attack, stroke, amputation and blindness 
if poorly managed or left untreated. In Type 1 
diabetes, which accounts for 10% of cases, the 
pancreas produces no insulin. Type 1 is not prevent- 
able and its cause remains unknown. The remain- 
ing 90% of people with the disease have Type 2 
diabetes, in which the pancreas does not produce 
enough insulin, or the body cannot properly use 
the insulin it does produce. Type 2 is most often 
preventable with lifestyle changes that include 
healthier eating and exercise. 

In recent years, diabetes, especially Type 2, has 
grown significantly as a health problem in Ontario, 
affecting the quality of life of people who have 
it and straining the health-care system. Factors 
driving this growth include high obesity rates, 
sedentary lifestyles, unhealthy diets and an aging 
population. Statistics from the Ministry of Health 
and Long-Term Care (Ministry) and the Canadian 
Diabetes Association suggest some alarming trends: 



• The number of people with diabetes in 
Ontario is expected to almost quadruple, 
from 546,000 in 2000 to 1.9 million by 2020, 
as illustrated in Figure 1. Ontario's diabetes 
growth rate is among the highest of all prov- 
inces, as illustrated in Figure 2. 

• People with diabetes incur medical costs that 
are roughly twice as high as those without 
the disease. A diabetes patient costs Ontario's 

Figure 1: Diabetes Prevalence and Costs in Ontario, 
2000-2020* 

Source of data: Canadian Diabetes Association 
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* Based on estimates by the Canadian Diabetes Association using trie 
Canadian Diabetes Cost Model. The two main sources of data used for 
the estimates and forecasts came from the National Diabetes Surveillance 
System and Health Canada's study "The Economic Burden of Illness in 
Canada." 
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Figure 2: Percentage Growth in Diabetes Prevalence, 
2000-2020* (%) 

Source of data: Canadian Diabetes Association 
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* Based on estimates by the Canadian Diabetes Association using the 
Canadian Diabetes Cost Model. The two main sources of data used for 
the estimates and forecasts came from the National Diabetes Surveillance 
System and Health Canada's study "The Economic Burden of Illness in 
Canada." 



health-care system more than $3,000 a 
year, but this can rise to more than $5,000 
if the patient experiences complications. 
Diabetes complications account for 69% of 
limb amputations, 53% of kidney dialysis and 
transplants, 39% of heart attacks and 35% 
of strokes. 

In 2008, the Ministry announced a new four- 
year, $741-million plan, called the Ontario Diabetes 
Strategy (Strategy), to expand services and improve 
the health of Ontarians with diabetes. The Ministry 
said that the Strategy's goals included raising 
awareness of diabetes risk factors through preven- 
tion programs, creating more Diabetes Education 
Teams to help patients better manage the disease, 
and developing an online Diabetes Registry to track 
individual patients and the overall prevalence of 
the disease across Ontario. The Ministry has identi- 
fied the Diabetes Registry as a "top clinical priority 
for eHealth Ontario," which has been working in 
partnership with a private-sector vendor to develop 
and implement it. 

According to the Canadian Diabetes Associa- 
tion, Ontario is one of five provinces with a formal 
diabetes strategy (the others are Alberta, Saskatch- 



ewan, Manitoba and Nova Scotia). In the four years 
up to the 2011/12 fiscal year, the Ministry allocated 
about $648 million of the Strategy's original fund- 
ing announcement of $741 million to various initia- 
tives, as illustrated in Figure 3. The Strategy was 
subsequently extended for another four years with 
a new funding approval of $152 million, which is in 
addition to funding for diabetes services delivered 
through other program areas of the Ministry. 



Audit Objective and Scope 



Our audit objective was to assess whether the 
Ministry of Health and Long-Term Care (Ministry) 
had adequate systems, policies and procedures in 
place to: 

• monitor and assess whether service providers 
are meeting the needs of people with diabetes 

Figure 3: Allocation of Ontario Diabetes Strategy 
Funding by Key Initiatives, 2008/09-2011/12 
($ million) 

Source of data: Ministry of Health and Long-Term Care 
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by providing them with timely access to 
appropriate and quality care; 

• ensure funding and resources provided for the 
Ontario Diabetes Strategy (Strategy) are used 
cost-effectively; and 

• measure and report periodically on the results 
and the effectiveness of the Strategy. 

In conducting our audit, we reviewed relevant 
policies and files, analyzed data, interviewed 
appropriate ministry staff and reviewed relevant 
studies from Ontario and other jurisdictions. We 
also interviewed staff and reviewed documents 
related to the Diabetes Registry at eHealth Ontario 
and Infrastructure Ontario. As well, we conducted 
surveys of about 10,500 physicians with assistance 
from the Ontario College of Family Physicians 
(580 responded) and of all 152 Diabetes Education 
Programs (DEPs), with the assistance of an external 
survey company (103 DEPs responded). In addi- 
tion, we contacted and visited stakeholders and dia- 
betes-care providers across the province, including 
six DEPs, five bariatric surgical sites, four Regional 
Coordination Centres, and the Canadian Diabetes 
Association. We also engaged an independent 
consultant with expert knowledge in the study of 
diabetes on an advisory basis. 

Our audit did not cover any initiatives under 
the Strategy that had just started, had recently 
changed, and/or had been audited by our Office in 
recent years. These include the Centres for Complex 
Diabetes Care (providing people with complex dia- 
betes a single point of access to specialized care), 
the Chronic Kidney Disease Program (providing 
dialysis services to diabetes patients who have kid- 
ney failure, a common diabetes complication), and 
the Insulin Pumps and Supplies Program (providing 
funding assistance to people with Type 1 diabetes) . 
As well, we did not rely on the Ministry's internal 
audit service to reduce the extent of our audit work 
because it had not recently conducted any audit 
work on diabetes initiatives in Ontario. 



Summary 



The number of people with diabetes in Ontario 
has more than doubled from 546,000 in 2000 to 
1.2 million in 2010, and that number is projected 
to grow to 1.9 million by 2020 — about one in 
every eight Ontarians. People with diabetes use 
the health-care system at about twice the rate of 
the general population, and the cost of diabetes to 
Ontario's health-care system is estimated to grow 
from $4.9 billion in 2010 to $7 billion by 2020. The 
Ministry recognized the long-term implications of 
this and in 2008 established a four-year $741 mil- 
lion Ontario Diabetes Strategy (Strategy). Although 
it is still too early to gauge the Strategy's mid- and 
long-term impact, in the short term the results have 
been mixed. 

On the one hand, there undoubtedly has been 
an improvement in the availability of diabetes 
care, giving people more options and knowledge to 
enable them to manage the impact of diabetes. On 
the other hand, most of the diabetes service pro- 
viders that were set up with Strategy funding are 
under-utilized, and many of those who responded 
to our surveys felt that more of their funding should 
be directed toward preventive services. 

According to the Diabetes Expert Panel estab- 
lished by the Ministry in 2006, "keeping people well 
and preventing disease is the most cost-effective, 
affordable and sustainable strategy for coping with 
chronic disease." We noted, however, that 97% of 
the $741 million funding was earmarked to treat 
people who already had diabetes, with only 3% for 
preventive initiatives. Given that 90% of people 
with diabetes have Type 2, which can often be 
prevented or postponed with good nutrition and 
exercise to limit weight gain along with other pre- 
ventive measures, we believe an increased focus on 
prevention warrants consideration by the Ministry. 

Some of our other observations were as follows: 

• eHealth Ontario (eHealth) has been working 
in partnership with a private-sector vendor to 
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develop and implement a new electronic Dia- 
betes Registry, a key initiative of the Strategy, 
to give physicians and the Ministry real-time 
patient data and comprehensive online mon- 
itoring of the disease. The Registry's original 
delivery date was April 2009, but this dead- 
line was not met and the proposed release 
date has been extended a number of times. 
Subsequent to our audit fieldwork, we were 
advised that the contract with the vendor was 
terminated in September 2012. 
In August 2010, eHealth and the vendor, 
through a request-for-proposal process, 
signed a six-year, $46-million contract that 
stipulated that the vendor would be paid only 
after successful completion of the Diabetes 
Registry. This was designed to help protect the 
public's interest and to motivate the vendor 
to deliver a system that meets performance 
requirements and timelines. Although no pay- 
ments had been made to the vendor as of mid- 
2012, the Ministry and eHealth had already 
incurred about $24.4 million in internal costs 
directly related to the Diabetes Registry since 
2008/09. They also spent another $50 mil- 
lion on other supporting electronic health 
records initiatives, such as electronic portals 
to other systems and the Ontario Laboratory 
Information System, which have already been 
deployed. eHealth acknowledged that the 
arrangement to pay the vendor only after suc- 
cessful completion of the contract has traded 
away much of the province's control over the 
project's design, progress and delivery time in 
exchange for price certainty. 
The province's 152 Diabetes Education Pro- 
grams (DEPs) help teach people with diabetes 
about the disease and how to manage it. Every 
DEP runs one or more Diabetes Education 
Teams (DETs), each consisting of a regis- 
tered nurse, a registered dietician and other 
professionals. DETs operate in hospitals, in 
community health centres and within Family 



Health Teams (FHTs) to educate diabetes 
patients. The Ministry funded 101 new DETs 
under the Strategy, increasing the total num- 
ber of DETs in Ontario to 322. However, many 
hospitals and FHTs have also set up education 
programs of their own with funding from 
other sources, including another branch of the 
Ministry, and this has led to service overlaps 
and under-utilization of about 90% of DEPs. 

• The DEPs are required to conduct audits regu- 
larly on the quality of care provided by their 
staff. However, the Ministry has never verified 
whether they actually do so. Our survey of 
DEPs found that about 25% said they "have 
not done" any audits or "do not know" they 
are required. 

• The Ministry provides an organization with 
$20 million annually to manage and fund 
diabetes service providers, including 47 DEPs 
in Northern Ontario, on its behalf. While 
the Ministry has an accountability agree- 
ment with the organization and a reporting 
process in place, it needs to significantly 
enhance its oversight of this organization 

to ensure that the organization and the 
service providers funded by it have used 
the Ministry's funding appropriately and 
in compliance with applicable policies. For 
instance, the organization has paid a consult- 
ing firm $105,000 since the 2009/10 fiscal 
year for "advice on election strategizing" and 
"developing relationships with relevant pol- 
itical decision-makers." We also noted that 
the organization could produce no original 
itemized receipts to support its staff meal 
expenses, contrary to its own policies and to 
policies set out by the government for provin- 
cially funded public-sector organizations. We 
found instances of staff claiming unreason- 
able amounts for meals and claiming alcohol, 
neither of which is in line with the rules of 
the Ontario Public Service. 

• The Ministry has significantly expanded 
Ontario's capacity for performing bariatric 
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surgery, a procedure that combats Type 2 
diabetes in obese people by removing part of 
the stomach and/or the small intestine. This 
has led to savings because far fewer out-of- 
country surgeries are being done. The number 
of in-province surgeries has risen significantly, 
from 245 in the 2007/08 fiscal year to 2,500 
in 2011/12. However, this still does not meet 
the current demand and is actually lower than 
the 2009/10 total of 2,900, when more out-of- 
country surgeries were performed. 

| OVERALL MINISTRY RESP0N! 

The Ministry acknowledges the recommenda- 
tions made by the Auditor General of Ontario 
and thanks him for conducting this timely audit. 
The Ministry and this government are commit- 
ted to the development and implementation of 
innovative initiatives and solutions that address 
the impact of diabetes and other chronic dis- 
eases on Ontarians. 

The Ministry welcomes the insights and 
recommendations provided by the Auditor Gen- 
eral. The audit identifies areas of consideration 
that the Ministry is already taking measures 
to address and reinforces the Ministry's com- 
mitment to addressing the complex challenge 
of diabetes. 

In June 2008, Cabinet approved $741 mil- 
lion in funding for the first four-year phase 
(2008-12) of the Ontario Diabetes Strategy, 
a comprehensive strategy to enhance preven- 
tion and management of diabetes and improve 
health outcomes for Ontarians impacted by 
diabetes. In April 2012, the Ministry confirmed 
continued funding for the Strategy of $152 mil- 
lion from 2012 to 2016. 

Through the Strategy, the government has 
built on existing investments in prevention and 
care across the health system to build capacity, 
make it easier for people to get services they 
need, and improve the overall quality of dia- 
betes service and care in Ontario. The result 



will be more opportunities to prevent diabetes 
for those at risk and, for people with diabetes, a 
more positive experience with the health system 
and a better quality of life. 

Investments in the Strategy will help build 
system capacity and an infrastructure for a com- 
prehensive chronic disease prevention and man- 
agement system that may be readily expanded 
to address other chronic conditions. 



Detailed Audit Observations 



KEY INITIATIVES AND PROGRESS OF 
ONTARIO DIABETES STRATEGY 

In response to Ontario's growing diabetic popula- 
tion and the costs associated with it, the govern- 
ment launched the Ontario Diabetes Strategy 
(Strategy) in the 2008/09 fiscal year with funding 
of $741 million over four years. The Ministry said at 
the time that the Strategy aimed to "prevent, man- 
age and treat diabetes across the province." 

The Ministry has used the modified Kaiser 
Chronic Disease Management model to graph- 
ically illustrate diabetes services along the 
continuum of care, as illustrated in Figure 4. 
The model divides Ontarians into three levels, 
depending on the severity of their diabetes, and a 
fourth for people who do not have the disease but 
are at risk of contracting it. The Ministry defined 
specific initiatives for dealing with each level, as 
well as "system enablers" to assist in executing 
the overall Strategy. Since the Strategy's introduc- 
tion, the Ministry has improved the availability 
of diabetes services by funding 101 new Diabetes 
Education Programs (DEPs) and establishing 14 
Regional Coordination Centres (RCCs) across 
the province. 

The Ministry also developed performance 
measures to assess the Strategy's progress at vari- 
ous stages. As Figure 5 illustrates, the short-term 
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Figure 4: The Modified Kaiser Chronic Disease Management Model 

Source of data: Ministry of Health and Long-Term Care 





Key Initiatives or Investments under Strategy 


Level 3 
Complex diabetes 

(-175,000*) 


• Bariatric surgery 

• Centres for Complex Diabetes Care (CCDCs) 

• Chronic Kidney Disease Program 

• Insulin Pump and Supplies Program 


Level 2 
Moderate diabetes 

(-409,000*) 

^fl Level 1 
^fl Early diabetes 

^fl (-584,000*) 


• Diabetes Education Programs (DEPs) 

• Self-management Initiative 

• Tools for individuals and health-care providers 
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• Prevention programs for high-risk groups (e.g., 
Aboriginal people, Asians) 

• EatRight Ontario 

• Improved access to primary care for prevention 
and early identification 


System enablers 


• Diabetes Registry and Baseline Diabetes Dataset 
Initiative (BDDI) 

• Regional Coordination Centres (RCCs) 



* Based on estimates of diabetes prevalence by the Canadian Diabetes Association. 



performance measures have shown mixed results 
and the intermediate or long-term measures have 
not yet shown improvement. In particular, hospi- 
talization rates for heart attacks, infections, ulcers 
or amputations among people with diabetes have 
increased and did not meet targets, but the Ministry 
indicated that it has not established specific time- 
lines for achieving these targets. Overall, rates have 
stabilized or decreased for some diabetes-related 
complications, hospitalizations and deaths, but the 
actual numbers remain high and can be expected to 
continue to rise significantly due to the growth of 
the diabetic population. 

Based on the data available to date, we noted 
that it is still too early to gauge the Strategy's 
impact over the medium and long term. Clearly, 
however, much remains to be done to reduce the 
growing burden of diabetes in Ontario. 



DIABETES REGISTRY AND BASELINE 
DIABETES DATASET INITIATIVE 

A key initiative of the Strategy was to have been the 
electronic Diabetes Registry, a database containing 
information about every Ontarian with diabetes 
intended to facilitate the delivery of care by clin- 
icians. The Registry, in conjunction with other 
foundational systems developed by eHealth Ontario 
(eHealth), an agency of the Ministry, would also 
be a first step toward an eventual province-wide 
Electronic Health Record for every Ontarian by 
2015. Pending delivery of the Registry, the Ministry 
implemented the Baseline Diabetes Dataset Initia- 
tive (BDDI) as an interim measure to provide phys- 
icians with paper reports containing their patients' 
most recent dates for the three key diabetes tests. 
The Ministry has continued to use the BDDI in the 
absence of a complete Diabetes Registry. 
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Figure 5: Summary of Key Performance Measures for the Ontario Diabetes Strategy 

Source of data: Ministry of Health and Long-Term Care 



Key Performance Measures for the Stra 




t Baseline 




Short-term Measures 



% of Ontarians who are physically inactive 


45 


52.6 


51.2 


51.0 


% of Ontarians who are overweight or obese 


_b 


51.8 


51.7 


53.0 


% of Ontarians with diabetes who have a regular family doctor 


_b 


97.6 


96.4 


96.9 


Number of Ontarians with diabetes registered with Health Care Connect 


_b 


3,744 


4,768 


10,335 


% of Ontarians with diabetes referred to family health-care provider by 
Health Care Connect 


_b 


60.5 


59.8 


63.0 


% of Ontarians with diabetes for whom a Diabetes Management 
Incentive (Q040) was billed 


_b 


25.5 


26.3 


29.2 


% of Ontarians with diabetes for whom a Diabetes Management 
Assessment (K030) was billed 


_b 


27.5 


28.5 


32.0 


% of Ontarians with diabetes who received all three key tests within the 
guideline periods 


80 


37.6 


37.8 


39.6 


Intermediate or Long-term Measures 


% of diabetes prevalence in Ontario population 


_b 


8.7 


9.3 


9.4 


Rate of emergency visits for hyperglycemia or hypoglycemia (abnormal 
blood-sugar levels) per 100,000 people with diabetes 


_b 


1,185 


1,115 


1,063 


Rate of kidney dialysis and transplant per 100,000 people with 
diabetes 


Maintain 
original rate 


712 


842 


845 


Rate of hospitalization for infections, ulcers or amputations per 
100,000 people with diabetes 


Reduce last 
reported rate 
by 10% 


2,294 


2,932 


3,347 


Rate of hospitalization for heart attacks per 100,000 people with 
diabetes 


Reduce last 
reported rate 
by 10% 


877 


1,022 


1,082 


Rate of eye surgeries per 100,000 people with diabetes 


_b 


3,500 


3,612 


3,365 



* Baseline data were derived between April 1, 2008, and March 31, 2010, depending on the performance measure. 

a. Although the Strategy did not set a target for this measure, the ACTIVE2010 Strategy, launched in 2004, set a target of increasing Ontario's physically active 
population to 55% by 2010 (or reducing its physically inactive population to 45% by 2010). 

b. The Strategy did not set a target for this measure. 



Diabetes Registry 

The Diabetes Registry is an interactive, real-time 
information system that health-care providers can 
use to quickly identify and manage Ontarians with 
diabetes, check patient records, access diagnostic 
information, send patient alerts and track the care 
provided to patients against evidence-based guide- 
lines. The Registry is intended to be used to per- 
form continuous and comprehensive province-wide 
surveillance of diabetes to support planning and 
monitoring of care. eHealth has been working in 



partnership with a private-sector vendor to develop 
and implement the Registry, which the Ministry 
identified as "a top clinical priority for eHealth." 

Timeline of Diabetes Registry Development 

The original target date for a first release of the 
Diabetes Registry was April 2009, but this deadline 
was not met. According to our review of documenta- 
tion — which included reports to the Management 
Board of Cabinet and the Treasury Board, the 
project charter and plan, original and amended 
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agreements, and meeting minutes between eHealth 
and the vendor — the proposed release date was 
moved forward a number of times. The Registry still 
had not been delivered at the end of our fieldwork 
in June 2012. 

Management at eHealth told us it could com- 
ment only on the timing changes that had taken 
place after April 2010, which was when the new 
management took office (eHealth changed its 
management after our 2009 audit of Ontario's 
Electronic Health Records Initiative) . eHealth 
also informed us that the official and contractual 
targeted release date under its current management 
was June 2011. 

On completion, the Diabetes Registry initially 
was to have been phased in at several pilot sites 
within two of the province's 14 Local Health 
Integration Networks (LHINs), the not-for-profit 
corporations that plan, integrate and fund local 
health services. However, the two pilot LHINs 
expressed concerns about the repeated delays and 
about the fact that eHealth no longer meets with 
them weekly to discuss the status of the Registry 
project. For its part, eHealth advised us that it and 
the LHINs decided in fall 2011 to suspend the meet- 
ings because of the lack of a reliable schedule for 
delivery of the Registry. Given that the first release 
of the Registry at the pilot sites will have only 
limited functionality, one of the two LHINs told 
us that it did not see a clear road map for future 
enhancements; the other questioned the value of 
the Registry altogether. 

eHealth cited the following reasons for the 
repeated delays: 

• The Treasury Board approved the Diabetes 
Registry budget in August 2008, but procure- 
ment was put on hold until May 2009 because 
of the creation of eHealth and a government 
decision to involve Infrastructure Ontario in 
selecting a vendor to develop and implement 
the Registry. Through a request-for-proposal 
process, eHealth signed the contract with the 
vendor in August 2010. 



• When bidding for the Diabetes Registry con- 
tract, the vendor who won the contract may 
have underestimated both the time required 
for the project and the project's complexity. 

• The vendor's project-management team may 
have failed to identify the project's critical path 
timelines and effectively manage completion 
of the project components and deliverables. 

• The project-design blueprint developed by 
the vendor appeared to contain many errors 
and omissions, which led to rejections and 
reworking of the design. 

• The vendor spent a great deal of time fix- 
ing numerous quality issues in the Diabetes 
Registry since the first test in September 2011. 
There were still hundreds of defects remain- 
ing in March 2012. 

Development Costs of Diabetes Registry 

In August 2010, eHealth signed a six-year, 
$46-million contract with a vendor to design, 
build, implement and manage the Diabetes Regis- 
try. The contract included a lump-sum payment 
of about $12 million when the Registry is ready 
for use and a monthly payment of $575,000 for 60 
months during a period of operation and mainten- 
ance. Payments to the vendor would not begin 
until after the Registry was successfully completed 
and deployed. 

Since no payments had been made to the vendor 
of the Registry as of June 2012, eHealth has not 
spent all of the $150 million funding approved 
under the Strategy for the Registry and other 
related eHealth projects. Since the 2008/09 fiscal 
year, the Ministry and eHealth have incurred sig- 
nificant internal costs to develop the Registry and 
other related projects. Specifically: 

• Since the 2009/10 fiscal year, eHealth has 
incurred about $20 million in costs for pro- 
jects directly related to the Diabetes Registry 
and another $50 million on other key elec- 
tronic health records initiatives. These include 
the electronic portals to other systems, the 
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Ontario Laboratory Information System, the 
Patient Registry and the Provider Registry 
which have been deployed and play a role in 
supporting improvements in diabetes care and 
management. 
• In the 2008/09 fiscal year, the Ministry 
engaged consultants to carry out Diabetes 
Registry-related projects at a total cost of 
about $4.4 million. The Ministry informed 
us that some of the projects were suspended 
due to the transition of the Registry from the 
Ministry to the newly formed eHealth, and to 
changes to eHealth's strategic direction fol- 
lowing recommendations of our 2009 audit of 
Ontario's Electronic Health Records Initiative. 
According to the contract with the vendor, it 
appeared that eHealth could terminate the contract 
without any obligation to make any payments if the 
vendor did not achieve the project's independently 
certified completion by a specified date. While the 
Diabetes Registry was regarded as a clinical priority 
for the implementation of certain key aspects of 
the Strategy, we were advised that with many more 
physicians migrating to Electronic Medical Records, 
the Registry is no longer seen as an essential com- 
ponent. Subsequent to completing our audit field- 
work, we were advised that the contract with the 
vendor had been terminated in September 2012. 

Procurement Process and Contract Management 

Our review of eHealth's procurement documents 
related to the Diabetes Registry in the 2009/10 fis- 
cal year indicated that most procurements did not 
follow an open competitive process but, instead, 
were sole-sourced without the approvals necessary 
to bypass the competitive process. However, sub- 
sequent to our 2009 audit of Ontario's Electronic 
Health Records Initiative, we noted that eHealth 
improved its procurement processes: all procure- 
ments we examined in the 2010/11 and 2011/12 
fiscal years followed an open competitive process. 

eHealth also informed us that it instituted a pro- 
ject management methodology to track deliverables 



and expenditures throughout the life cycle of a 
project to give management a snapshot perspective 
of a project's progress at anytime. However, this 
methodology did not apply to the Registry, which 
was developed using the Alternative Financing and 
Procurement (AFP) model, the first time that the 
government used AFP for an information technol- 
ogy project. eHealth advised us that it applied an 
alternative project oversight methodology specific 
to an AFP technology-related project to the Dia- 
betes Registry. 

Under AFP, the government as client establishes 
the scope and purpose of the project, and the work 
is financed and carried out by a private-sector 
vendor. Only after a project is complete does the 
private-sector vendor get paid. According to Infra- 
structure Ontario, which provided eHealth with 
procurement-management services for the Registry 
project, AFP allows for projects to be delivered 
faster and more efficiently, on time and on budget. 
AFP also differs from traditional procurement pro- 
cesses in that it is designed to avoid cost overruns 
and to transfer risks from the government to the 
vendor, who has the expertise and experience to 
handle them. 

eHealth acknowledged that there have been 
trade-offs in using AFP for the Registry; as the 
vendor gets paid only upon successful completion 
and delivery of the project, the province gets price 
certainty in exchange for only minimal influence on 
the design, progress and delivery time of the pro- 
ject. Thus, while the province does not have to pay 
the vendor for non-delivery of the Registry, it still 
ultimately bears other risks and costs associated 
with project delays. 

Infrastructure Ontario engaged an external firm 
to perform a risk analysis before eHealth signed the 
contract with the vendor, and the analysis identi- 
fied the risk of delays, which could increase costs 
and discourage user adoption. However, Infrastruc- 
ture Ontario concluded that the stipulation that the 
vendor would not be paid until after the system was 
certified as complete would be a sufficient incentive 
to ensure that the vendor completed the project. 
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Infrastructure Ontario engaged an Independent 
Certifier on behalf of eHealth and the vendor to 
monitor the Registry's progress and to issue a final 
acceptance. In its monthly reports, the Certifier 
repeatedly raised concerns about impediments to 
its ability to assess, measure or control the project's 
progress. However, while eHealth had similar 
concerns, it informed us that under AFP, it had less 
leverage to require the vendor to address the defi- 
ciencies identified by the Certifier, including: 

• serious delays with the overall project; 

• no visibility into the project's schedule, scope 
and progress, and none into the causes for 
the delays; 

• absence of an approved project plan and no 
targets for milestones; 

• tight timeline with minimum buffer; 

• quality concerns, as evidenced by the large 
number of defects identified; and 

• fragmented testing approach, with multiple 
plans and ambiguous criteria. 

In April 2011, in an attempt to improve the 
timeliness and certainty of delivery of the Diabetes 
Registry, the project was split into two releases: one 
to be delivered in September 2011 and the other 
in May 2012. The vendor was to be paid 75% of 
the acceptance payment after the first release and 
the remaining 25% after the second, as opposed to 
100% on successful completion of the entire Regis- 
try as per the original contract. 

eHealth informed us that the purpose of amend- 
ing the contract was to ensure that the first release, 
a core clinical module, was delivered as early as 
possible. However, at the end of our audit fieldwork 
in June 2012, neither release was ready and the 
delivery date remained uncertain. Infrastructure 
Ontario advised us that, in its opinion, making first 
payment to the vendor only after project comple- 
tion constitutes an incentive for the vendor to com- 
plete and deliver the project as soon as possible. 
As well, the Ministry and eHealth informed us that 
they have taken steps to mitigate risks associated 
with delays by continuing to develop other comple- 
mentary initiatives that support diabetes care and 
management. 



Baseline Diabetes Dataset Initiative 

The Ministry launched the Baseline Diabetes Data- 
set Initiative (BDDI) in spring 2009 as an interim 
measure pending delivery of the Registry. The BDDI 
provides physicians with paper reports containing 
information on their diabetes patients' most recent 
dates for three key tests. This information helps 
physicians to better manage their diabetes patients 
by determining the dates on which patients should 
receive their next tests. The recommended time 
frame for each test is as follows: 

• blood-sugar test at least once every six months; 

• cholesterol test at least once a year; and 

• retinal eye exam at least once every two years. 
The Ministry and eHealth spent $5.6 million 

between 2009 and 2012 to implement three waves 
of the BDDI and another $2.6 million in incen- 
tive payments to physicians to encourage them to 
participate in it. Although the Canadian Diabetes 
Association has cited the BDDI as a best practice 
that "facilitates identification of patients for testing 
on the three tests," we noted the following: 

• The Ministry set a long-term goal that 80% 
of adult Ontarians with diabetes should 
receive all three key tests within the recom- 
mended times. According to BDDI results as of 
December 31, 2011, 45.6% of adult Ontarians 
with diabetes had all three tests within the 
recommended periods — still well below the 
long-term target of 80% and only slightly 
higher than on December 31, 2009 (43.1%). 
The Ministry informed us there is no pre- 
determined timeline for achieving this target. 

• The BDDI does not include test results from 
hospital labs, even though hospitals accounted 
for about one-third of all labs in Ontario and 
performed 9% of blood-sugar tests and 10% of 
cholesterol tests. According to our physician 
survey and our review of correspondence 
between the physicians and the Ministry, many 
physicians indicated that hospital labs are 

the only major lab services in some rural and 
northern regions. The Ministry acknowledged 
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this limitation but noted that the BDDI is 
an interim measure pending delivery of the 
Registry. 

Many of the physicians who responded to our 
survey and the four Regional Coordination 
Centres we visited expressed other concerns 
about the BDDI, and a number questioned its 
usefulness. They noted that the information 
provided by the BDDI was not timely and was 
already accessible through currently avail- 
able Electronic Medical Records now being 
used by 43% of Ontario physicians. They also 
noted that the BDDI was a time-consuming 
manual process of sharing and reviewing 
patient information. 

Participation in the BDDI is voluntary. As of 
April 2012, only about half of Ontario phys- 
icians were taking part in it. 



RECOMMENDATION 1 



To allow for efficient and effective diabetes sur- 
veillance at the provincial level and to gauge the 
progress of the Ontario Diabetes Strategy, the 
Ministry of Health and Long-Term Care (Min- 
istry) should work closely with eHealth Ontario 
(eHealth) and Infrastructure Ontario to: 

• ensure that eHealth's initiatives for chronic- 
disease prevention and management are 
implemented with an appropriate quality 
assurance process so that they meet the 
needs of physicians and other users; and 

• implement measures based on lessons 
learned from using the total outsourcing 
system development model for the Diabetes 
Registry if this procurement process is used 
for future information technology projects. 

| MINISTRY RESPON! 

Quality assurance with respect to electronic 
health tools has been, and will continue to be, 
a priority of both the Ministry and eHealth. 
Despite the challenges the vendor has faced 
with the Diabetes Registry, eHealth has made 



significant advances in meeting its obligations 
to develop, integrate and deliver the comple- 
mentary foundational systems that serve as the 
backbone of Electronic Health Records. 

In applying lessons learned from the Dia- 
betes Registry procurement model, eHealth 
has already taken steps to improve the procure- 
ment process for future projects. In particular, 
eHealth has applied the AFP model's milestone- 
based payment structure to procurement of the 
Drug Information System and has combined this 
with a rigorous procurement process with con- 
tractual provisions allowing eHealth to closely 
monitor progress, approve the vendor's delivery 
strategy and hold the vendor more accountable 
for delays. 



DIABETES PREVENTION AND 
HEALTH PROMOTION 

Type 1 diabetes is not preventable. But the opposite 
is true for Type 2, which accounts for 90% of the 
diabetic population. The World Health Organiza- 
tion (WHO) says a healthy high-fibre and low-fat 
diet, along with at least 30 minutes a day of 
physical activity, can reduce the risk of Type 2 dia- 
betes by 50%. The WHO also notes that more than 
90% of Type 2 cases can be prevented or postponed 
with good nutrition, regular physical activity, smok- 
ing cessation and effective stress management. 

According to the Diabetes Expert Panel estab- 
lished by the Ministry in 2006, "keeping people well 
and preventing disease is the most cost-effective, 
affordable, and sustainable strategy for coping with 
chronic disease." The Canadian Diabetes Associa- 
tion 2008 Clinical Practice Guidelines also noted 
that "preventing Type 2 diabetes would result in 
significant public health benefits, including lower 
rates of cardiovascular disease, renal failure, blind- 
ness, and premature mortality." 
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Coverage of Diabetes Prevention Initiatives 

The Ministry has earmarked only 3% of the 
$648 million funding allocation of the Ontario Dia- 
betes Strategy (Strategy) — about $19 million — for 
prevention (see Figure 3) . The Ministry informed 
us that this funding for diabetes prevention initia- 
tives under the Strategy focused on reducing the 
risks of new diabetes cases. The funding targeted 
prevention initiatives at 24 local agencies in selected 
communities with high prevalence of diabetes and 
large concentrations of ethnic groups at high risk 
of developing diabetes (for example, Asians and 
Aboriginal people) . We noted that: 

• Prevention initiatives in the Strategy targeted 
the regions of Toronto, Peel, North East and 
North West, even though statistics from the 
Institute for Clinical Evaluation Sciences, a 
leading independent research organization 
in Canada, noted that other Local Health 
Integration Network (LHIN) regions, such as 
Central East and Erie St. Clair, also have an 
equally high prevalence of diabetes. 

• The Ministry achieved its target of reach- 
ing 40,000 at-risk people in communities 
with a high prevalence of diabetes through 
community-based primary prevention initia- 
tives but acknowledged that it has little reliable 
information about the size and distribution of 
the entire at-risk population in Ontario. 

• Almost two-thirds of Diabetes Education Pro- 
grams that responded to our survey said they 
thought more of their resources should go to 
educating people at high risk of developing 
diabetes. However, they have been unable 

to do this because their funding from the 
Ministry is intended to assist people already 
diagnosed with diabetes. 

Effectiveness of Diabetes 
Prevention Initiatives 

Policy makers need reliable evaluations of the 
effectiveness of prevention initiatives to help them 



set priorities. Such evaluations involve ongoing 
quantitative analysis of the impact of preven- 
tion initiatives on mitigating the risk factors for 
diabetes, which include unhealthy diet, physical 
inactivity, and being overweight or obese. 

The Ministry informed us that it developed no 
performance measure under the Strategy to mon- 
itor the extent of unhealthy eating among Ontar- 
ians because scientific literature has not established 
unhealthy diet as an independent causal risk factor 
for diabetes. It has monitored two other risk fac- 
tors, body weight and physical inactivity. Although 
the Ministry has not developed targets for these 
two measures as part of the Strategy, it said that 
its ACTIVE2010 Strategy in 2004 set a target of 
reducing the proportion of the population that is 
physically inactive to 45% from 52% by 2010. 

However, as shown in Figure 5, these two meas- 
ures have remained relatively constant in recent 
years. More specifically, the October 2011 Report 
of the Strategy Key Performance Measures said 
that "the proportion of adults who are overweight 
or obese has increased significantly since 2003/04 
(from 49.6% to 53.0%)" and "the proportion of 
physically inactive adults has not improved since 
2003/04, fluctuating from 50% to 53%." 

In May 2011, the Ministry commissioned an 
external evaluation of its diabetes prevention 
initiatives. The evaluation concluded that the 
initiatives had limited impact on increasing public 
awareness of diabetes and on changing behaviours. 
In response, the Ministry conducted a diabetes 
forum in January 2012 with the local agencies that 
it funds to do diabetes prevention. The agencies 
had two major suggestions, which they say are 
needed to promote the long-term success of the 
prevention initiatives: 

• The Ministry needs to provide agencies with 
sustained multi-year funding commitments 
and avoid delays in transfer of time-limited 
funds to minimize implementation delays and 
difficulties in hiring and retaining staff. 

• The Ministry needs to improve its co-ordina- 
tion and communication with the agencies. 
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The agencies indicated that they have been 
"working in silos." In order to avoid duplicate 
efforts, the agencies suggested that the 
Ministry needs to take a more proactive role 
in developing "a centralized understanding 
of who's doing what, who needs additional 
help and support" and facilitating knowledge 
exchange on best practices. 
The Ministry informed us that it planned to 
conduct a further evaluation of its diabetes preven- 
tion initiatives during the 2012/13 fiscal year, 
and planned to use the results of this evaluation 
together with information from the diabetes forum 
to make changes to the prevention initiatives. 

Screening for Undiagnosed Diabetes 

People with undiagnosed diabetes are those who 
have developed the disease but have not yet been 
identified as such by health-care providers. Accord- 
ing to the Canadian Diabetes Association (Associa- 
tion) 2008 Clinical Practice Guidelines (Guidelines), 
undiagnosed Type 2 diabetes may occur in nearly 
3% of the adult population. So at least 284,000 
Ontarians may not know that they have diabetes. 
Studies indicate that Type 2 diabetes can remain 
without symptoms for up to 10 years, and that at 
the time of diagnosis, 20% to 30% of patients will 
have already developed complications. Undiagnosed 
diabetes, therefore, will place more costly burdens 
on the health-care system in the long run. 

Since 2008, the Association has recommended 
screening individuals as early as age 40 because 
this has proved useful in detecting undiagnosed 
diabetes. As yet, there has been no specific strategy 
for identifying undiagnosed cases across Ontario 
through targeted screening for Type 2 diabetes and 
pre-diabetes (a blood-sugar level that is higher than 
normal, but not yet high enough to be diagnosed as 
Type 2 diabetes) . 

The Ministry indicated that, to date, its focus has 
been on screening for people with risk factors for 
developing diabetes rather than identifying undiag- 
nosed diabetes. In that regard, the Ministry advised 



us that it has screened about 4,600 people for 
diabetic risk factors through its community-based 
prevention initiatives. It has also implemented a 
project called the Ottawa Model for Undiagnosed 
Diabetes aimed at identifying undiagnosed diabetes 
in patients hospitalized for other reasons and con- 
necting them to diabetes care in their community. 
The Ministry informed us that because this project 
covered only about 500 patients at four hospital 
sites, it would be expanded to other hospital sites in 
the 2012/13 fiscal year. 

Development of a Comprehensive Health- 
promotion Strategy 

The World Health Organization says a healthy 
lifestyle that includes healthy eating and physical 
activity is a first line of defence in the prevention 
of Type 2 diabetes. In 2005, the government estab- 
lished the Ministry of Health Promotion and Sport 
(which was merged into the Ministry of Health 
and Long-Term Care in October 2011). In 2006, 
the Ministry introduced Ontario's Action Plan for 
Healthy Eating and Active Living (HEAL) . However, 
as noted previously, there has been no recent 
improvement in the percentages of Ontarians who 
are overweight, obese or physically inactive. 

In March 2011, the government established 
the Commission on the Reform of Ontario's Public 
Services to provide advice on the way government 
delivers services. In February 2012, the Commission 
released a report suggesting the government should: 

• do more to promote healthy lifestyles; 

• establish a province-wide chronic-disease 
prevention strategy; 

• take a more comprehensive approach to popu- 
lation health; 

• explore regulatory options for the food 
industry; 

• work with the federal government on nutri- 
tion regulation; and 

• replicate British Columbia's ActNow initiative. 
We noted that several well-respected organiza- 
tions, including the WHO, the Conference Board of 
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Canada, and the Health Council of Canada, have 
also credited British Columbia's ActNow initia- 
tive, with its cross-government and multi-sectoral 
approach, as a best practice in health promotion 
and prevention of chronic disease. 

We also noted that in April 2012, Cancer Care 
Ontario and Public Health Ontario published 
a report indicating that Ontario's health-care 
spending focused on treating people after they 
became ill rather than on keeping them healthy 
in the first place. The report noted that there are 
four key risk factors — unhealthy eating, physical 
inactivity, tobacco use and alcohol consumption — 
that are strongly related to people developing a 
chronic disease such as diabetes and that Ontario 
has developed a comprehensive strategy only to 
reduce tobacco use. The report recommended a 
comprehensive chronic-disease-prevention strategy 
targeting the entire Ontario population using a 
whole-of-government approach that engages all 
sectors and levels of government, community 
groups, businesses, educational institutions and 
media. The Diabetes Education Programs and many 
of the physicians who responded to our surveys, as 
well as the diabetes experts we interviewed, also 
mentioned that Ontario lacks a multi-faceted strat- 
egy to prevent chronic disease and obesity. 

The Ministry informed us that the government 
has introduced initiatives to support healthy eating, 
including the Healthy Communities Fund Grant 
Program for the delivery of health-promotion 
initiatives; the Student Nutrition Program, funded 
by the Ministry of Children and Youth Services; and 
the Northern Fruit and Vegetable Program, target- 
ing elementary-school children in selected areas in 
Northern Ontario. Also, in May 2012, the govern- 
ment assembled a Healthy Kids Panel to advise it 
on the development of a childhood obesity strategy. 
We noted, however, that the adult population could 
be helped by similar health-promotion initiatives 
and a specific obesity strategy. 



RECOMMENDATION 2 



To enhance the focus on prevention and early 
detection of diabetes as long-term, cost-effective 
strategies, the Ministry of Health and Long- 
Term Care should: 

• re-assess whether allocating only 3% of total 
dedicated diabetes funding to prevention 
initiatives is the most cost-effective long- 
term strategy; 

• devise ways to identify, on a more timely 
basis, people with undiagnosed diabetes; and 

• develop comprehensive health-promotion 
strategies that focus on all Ontarians and 
consider similar strategies used in other 
jurisdictions. 

MINISTRY RESPONSI 

In addition to the allocation under the Ontario 
Diabetes Strategy, the Ministry has invested 
more than $335 million annually in health- 
promotion programs to promote healthy eating, 
prevent chronic diseases like diabetes, and 
reduce injury and addiction. However, the 
Ministry will undertake a review of the funding 
allocation for diabetes prevention initiatives 
under the Strategy. 

The Ministry is also currently developing 
a provincial framework that will inform a co- 
ordinated approach to diabetes screening in 
community-based primary health care organiza- 
tions across Ontario. 

The Ministry will continue to assess the com- 
prehensive health-promotion and prevention 
strategies used in other jurisdictions. The Min- 
istry currently participates in several federal/ 
provincial/territorial committees dealing with 
issues such as healthy weights, sodium reduc- 
tion and tobacco control, to share information 
on best practices. The integration of health pro- 
motion within the Ministry will allow a renewed 
focus on broader health-promotion and diabetes 
prevention strategies. 
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DIABETES EDUCATION PROGRAMS 

In recognition of the fact that providing appropri- 
ate information to people with diabetes is essential 
to the management of the disease, the government 
initiated the Diabetes Education Programs (DEPs) 
in 1992 and placed them in three settings: hospi- 
tals, community health centres and Family Health 
Teams. Each DEP consists of one or more Diabetes 
Education Teams (Teams), and each Team consists 
of a registered nurse and a registered dietician. 
Teams may also have other health-care profession- 
als, such as social workers, psychologists, foot-care 
specialists, pharmacists and physiotherapists. 
DEPs use counselling in groups and one-on-one to 
promote self-care and help patients improve their 
quality of life, minimize their symptoms, and pre- 
vent complications. 

Given the extra funding available with the Strat- 
egy, the Ministry has expanded the DEPs by adding 
101 new Teams since the 2009/10 fiscal year. Cur- 
rently, the Ministry funds 322 Teams in 152 DEPs 
across the province. Of the 152 DEPs, 47 in North- 
ern Ontario were funded and managed on behalf 
of the Ministry by a not-for-profit organization that 
has signed an accountability agreement with and 
received funding from the Ministry. 

Monitoring of Diabetes 
Education Programs 

Performance Evaluation of DEPs: 
Caseload Benchmark 

The Ministry has monitored the DEPs using a case- 
load benchmark: each Team (one full-time regis- 
tered nurse and one full-time registered dietician) 
in a DEP should have an active caseload of 1,000 
patients or more per fiscal year. We noted, however, 
that this benchmark may no longer be representa- 
tive because it was developed in 2001, when there 
were only 71 DEPs, compared to the current total 
of 152. The program's scope has also grown and 
evolved since 2001. 



As well, the benchmark was developed on the 
assumptions that each patient would get five hours 
of services a year, and that 60% of services would 
be delivered through individual counselling and 
40% in group sessions. However, the majority of 
the DEPs that responded to our survey indicated 
that these assumptions did not reflect their actual 
activities and needed to be reviewed. The Ministry 
informed us that it has initiated a benchmark 
review during our audit and was in the process of 
finalizing an analysis report in August 2012. 

We noted that about 90% of DEPs did not meet 
the 1,000-patient caseload benchmark in each 
fiscal year since 2008/09, and more than one- 
third of them failed to achieve even 50% of the 
benchmark in 2010/11. The Ministry informed us 
that in June 2011 it required any DEP that failed 
to meet at least 50% of the benchmark to submit 
a Performance Improvement Plan. After our audit 
fieldwork, the Ministry indicated that it was in 
the process of reviewing the results of those DEPs 
implementing the plans as part of its ongoing 
monitoring of the DEPs. 

Consistency and Quality of DEPs 

We noted that the Ministry has not adequately mon- 
itored the consistency and the quality of service 
provided by the DEPs across the province, other 
than to evaluate them against the 1,000-patient 
caseload benchmark. For example: 

• According to the Policies and Procedures 
Manual for DEPs issued by the Ministry, DEPs 
are required to conduct routine audits of the 
quality of care they provide on a regular basis 
and hand over all information about such 
audits to the Ministry on request. The Ministry 
has given the DEPs no guidance on what the 
audits should cover or how often they should 
be conducted; nor has the Ministry requested 
any audit results from DEPs to determine if 
they were in compliance with the require- 
ment. According to our survey of DEPs, the 
frequency of such audits varied, with about 
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one-quarter of the DEPs saying they "have not 
done" any audits or "do not know" they are 
required to conduct such audits. 

• Physicians who responded to our survey indi- 
cated that high staff turnover and inadequate 
training for new staff at DEPs has affected 
quality of care. In addition, they were uncer- 
tain about the DEPs' ability to give patients 
appropriate advice. They also noted that DEPs 
often promoted new drugs that had no proven 
track record, and provided education of vary- 
ing quality to patients. 

• There is a certification program for the desig- 
nation of Certified Diabetes Educator (CDE). 
However, since this program is voluntary, not 
every DEP has staff with the CDE designa- 
tion. The Ministry informed us that it has 
encouraged the DEPs to recruit staff with 
this certification and/or help staff obtain this 
certification. However, it has not tracked or 
monitored the competence, skill levels and 
qualifications of DEP staff. 

Oversight of Northern Diabetes 
Education Programs 

The Ministry has an accountability agreement with 
and provides transfer payments to a not-for-profit 
corporation to fund and manage 47 adult DEPs in 
Northern Ontario, 35 pediatric diabetes programs, 
Northern Ontario aboriginal diabetes services, and 
Regional Coordination Centres in the northeast 
and northwest regions. The Ministry informed us 
that the organization is responsible for selecting 
diabetes service-providers to meet its funded 
objectives as outlined in its accountability agree- 
ment with the Ministry. 

The organization is small, with about 10 staff 
located in three offices. The same senior manage- 
ment has been in place since its creation in 1992. 
Over the past decade, annual ministry funding to 
the organization has increased from $7 million to 
$20 million for expanding services across Northern 
Ontario. According to its accountability agreement 



with the Ministry, the organization is responsible 
for administering the funds in a prudent and 
effective way. 

The organization transfers most of its ministry 
funding to the community-based diabetes service 
providers, including the DEPs in the north, to 
deliver diabetes care. The organization has given 
about $66 million to service providers between the 
2006/07 and 2010/11 fiscal years, and transferred 
the majority of these funds, about $44.5 million, to 
adult DEPs. The Ministry requires the organization 
to have an accountability agreement with each 
northern DEP, and to collect quarterly and annual 
reports summarizing caseloads, activities and 
financial information from each DEP. The Ministry 
informed us that the organization has accounted 
for funding granted to the DEPs through an annual 
settlement and reconciliation process. However, our 
review of the organization's audited financial state- 
ments indicated that, since 2007, its auditors have 
been unable to provide a normal "clean" audit opin- 
ion of its financial statements because the auditors 
did not examine the records of the service providers 
funded by the organization. The organization 
informed us that it assumed the funds it gave the 
service providers went to providing diabetes care. 

Based on our review of the organization's 
expenditures, we also noted some questionable 
practices, some of which ran contrary to its own 
policies and/or to those in the Broader Public Sec- 
tor (BPS) Expense Directive that became effective 
April 1, 2011. Examples included: 

• Since the 2009/10 fiscal year, the organ- 
ization has paid a consulting firm a total 
of $105,000, including a $5,000 monthly 
"retainer fee" and reimbursements for 
telephone, fax, meal and taxi expenses. In 
October 2010, it signed a formal agreement 
with the firm for such services as "providing 
strategic advice on election strategizing and 
membership mobilization," and "developing 
relationships with relevant political and 
bureaucratic decision-makers." The organiza- 
tion informed us that its board of directors 
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approved the engagement of the consulting 
firm for high-level strategic advice to ensure 
that the government was aware of the organ- 
ization. However, the Ministry informed us 
that it never approved a budget for consulting 
services of this nature and that the organiza- 
tion never sought ministry approval to reallo- 
cate funding to these services. 
The organization had inadequate record- 
keeping practices relating to staff travel and 
meal expenses. Almost all of the meal expenses 
we reviewed were supported only by credit- 
card slips or handwritten notes rather than by 
the original itemized and detailed receipts, as 
required by the organization's own policies and 
the BPS Expense Directive. Often, these records 
did not specify the names of the people who 
attended or the purpose of the meals. 
The organization's policies did not align 
with the rules in the Ontario Public Service 
(OPS) regarding consumption of alcohol and 
reasonableness of meal expenses. Although 
the organization informed us that it did not 
encourage the claiming of alcohol on expense 
charges, we found instances where staff did 
so. As well, the organization's policies did 
not specify the maximum amounts per meal. 
We also noted instances where staff claimed 
meals costing up to $80 per person. 
Travel expenses claimed by the organization's 
CEO amounted to about $40,000 each year. 
When travelling to Toronto, the CEO received 
$100 per day for accommodation in a private 
home. This was four times more than the 
per diem rate of $25 specified in its policies. 
Although the BPS Expense Directive does not 
specifically address this per diem, it does say 
that "due to mandatory requirements, and the 
principles of transparency and accountability, 
it is clear that per diems would no longer be 
allowed" and "in the OPS, per diems are no 
longer used." 

The organization leased an office in Kenora 
in October 2010 but decided to close it in July 



2011 due to difficulties in securing qualified 
staff. This resulted in a lease termination 
charge of about $12,000. The Ministry 
informed us that although the organiza- 
tion had discussed with it the consolidation 
of its offices in Kenora and Thunder Bay, 
the Ministry was unaware that the Kenora 
office had been closed and that the organ- 
ization had incurred a relatively significant 
termination charge. 
• The organization paid about $2,000 to send 
an executive assistant to Las Vegas for a 
management course designed for managers, 
supervisors and others with management 
responsibilities. 
While we suggested that an enhanced oversight 
of this organization was required, the Ministry indi- 
cated that, as with any broader-public-sector organ- 
ization it funds, its oversight role does not include 
monitoring expenses incurred by the organization's 
staff. The Ministry said it expected organizations to 
have proper policies and procedures in place, but 
indicated that ensuring adherence to the applicable 
policies and directives is the responsibility of the 
organization's board of directors. 

Subsequent to our audit, the Ministry informed 
us that it had taken some actions, including the 
recovery of about $40,000 in unapproved expendi- 
tures from the organization in June 2012. The 
Ministry also followed up in areas of the organiza- 
tion's non-compliance with requirements of the 
Broader Public Sector Accountability Agreement 
in August 2012. 



RECOMMENDATION 3 



To ensure that Diabetes Education Programs 
(DEPs) provide diabetes patients with consistent 
and quality care, and in compliance with applic- 
able policies, the Ministry of Health and Long- 
Term Care should strengthen its oversight of 
DEPs and other recipients of diabetes funding by: 
• developing appropriate service-delivery and 
cost-effectiveness measures and requiring 
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DEPs to periodically report on these meas- 
ures; and 

• conducting periodic site visits to selected 
regional, community and broader- 
public-sector organizations that receive 
diabetes funding. 

| MINISTRY RESPOND 

The Ministry's current agreements with transfer- 
payment recipients require that local diabetes 
programs comply with and provide minimum 
service components. The Ministry together with 
the LHINs, will conduct a follow-up review of 
the agreement framework to identify further 
opportunities to strengthen accountability. 

Apart from the measurement framework 
developed by the Ministry in partnership 
with Health Quality Ontario and the Regional 
Coordination Centres in 2011/12, the Ministry 
will develop additional DEP performance 
measures and reporting requirements that are 
aligned with the attributes defined in the meas- 
urement framework. 

The Ministry will review how it can apply 
greater oversight of broader public sector organ- 
izations receiving funding for diabetes services, 
and will continue to take timely, appropriate 
action when non-compliance with agreements 
is identified. The Ministry is developing further 
monitoring activities such as refinements to 
its annual reconciliation process, developing a 
protocol for periodic site visits, and examining 
the potential of obtaining attestations from the 
organizations' boards to ensure their compli- 
ance with the requirements of the Transfer Pay- 
ment Accountability Agreements. 



CO-ORDINATION OF AND ACCESS TO 
DIABETES-CARE PROVIDERS 

Co-ordination among DEPs 

As part of the Strategy, the Ministry's Provincial 
Program Branch (PPB) has funded 101 new DEP 
teams in hospitals, community health centres and 
Family Health Teams (FHTs) since the 2009/10 
fiscal year. The Ministry informed us that the loca- 
tions of the new teams were based on proposals 
from the LHINs. At the same time, we noted that 
the Ministry also funded other diabetes programs. 
For example, FHTs have received funding from the 
Ministry's Primary Care Branch to set up chronic- 
disease management programs, which include 
diabetes programs, and hospitals have allocated 
portions of their global funding to set up diabetes 
programs. However, the Ministry informed us that 
the PPB maintained records only for PPB-funded 
DEPs and was unable to confirm which other FHTs 
and hospitals have also funded diabetes services. 

The DEPs we visited, and those that responded 
to our survey, indicated that they have been unable 
to meet the Ministry's 1,000-patient caseload 
benchmark because their catchment areas over- 
lapped with those of other diabetes programs. This 
situation has led to under-utilization of many DEPs 
and even competition among DEPs for diabetes 
patients as they attempt to meet the benchmark. In 
fact, all of the DEPs we visited were located close to 
other diabetes programs. For example: 

• One DEP was located within a five-minute 
walk of two FHTs with diabetes programs and 
within a 10-minute drive of another DEP. 

• A DEP was located in the same hospital as a 
registered dietician who also delivered educa- 
tion about diabetes management in a hospital- 
based clinic. 

• One significantly under-utilized DEP was 
located in a rural area that also had four other 
DEPs covering the same catchment area. 
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Co-ordination between DEPs and 
Physicians 

The Canadian Diabetes Association 2008 Clinical 
Practice Guidelines recommended diabetes self- 
management education for all people with diabetes. 
In 2011, a team of researchers from two Ontario 
universities studied physicians' patterns of refer- 
ring their patients to DEPs in a suburban region 
of southern Ontario. With the assistance of the 
Ontario College of Family Physicians, we extended 
this study by surveying all family physicians in the 
province. Both the 2011 study and our survey found 
that many physicians did not refer all of their dia- 
betes patients to DEPs. The most common reasons 
given for not referring were "patients unwilling 
to go" and "physicians able to provide education 
in-house." The main reasons that patients were 
unwilling to go were "times unsuitable, with no 
evening or weekend services" and "language bar- 
riers." These reasons were in line with our survey 
of DEPs, which found that almost all of them were 
closed evenings and weekends, and about half did 
not offer language-specific services. 

As well, our DEP and physician surveys 
suggested that lack of communication and co- 
ordination between the DEPs and physicians has 
been a barrier to provision of diabetes care. About 
60% of the DEPs that responded to our survey 
noted that co-ordination and communication with 
other health-care providers, especially physicians, 
has been a major challenge. Even though lab test 
results on blood sugar and cholesterol could help 
the DEPs monitor their patients, 43% of the DEPs 
in our survey said they received lab test results 
only half the time or less from physicians and other 
health-care providers. Physicians, for their part, 
noted they often did not receive information from 
local DEPs regarding the progress of patients the 
DEPs were seeing. 



Diabetes Management Incentive 
for Physicians 

In April 2006, the Ministry introduced a Diabetes 
Management Incentive (DMI) to promote qual- 
ity diabetes care. DMI is a $75 annual payment 
to physicians for co-ordinating, providing and 
documenting all required elements of care for each 
diabetes patient consistent with the Canadian Dia- 
betes Association 2008 Clinical Practice Guidelines. 
Ontario Health Insurance Plan (OHIP) data showed 
that the Ministry made about $95 million in DMI 
payments to physicians since 2006. However, we 
noted that the impact of the DMI in encouraging 
physicians to provide continuous and co-ordinated 
diabetes management was unclear. Specifically: 

• Forty-six per cent of physicians who 
responded to our survey indicated that the 
DMI had no impact on the way they managed 
their patients. Of these, about 14% did not 
even realize the DMI existed. 

• The October 2011 report on the Ontario 
Diabetes Strategy Key Performance Measures 
showed that the DMI was claimed for less 
than 30% of diabetes patients in each fiscal 
year since 2008/09, with only a slight increase 
during the period, as illustrated in Figure 5. 
This suggested a lack of awareness or usage of 
the DMI. 

• In April 2012, researchers from the University 
of Toronto released a study reporting little 
difference in overall performance before 

and after the introduction of the DMI. They 
found that those physicians already providing 
higher-quality care simply continued to do so 
while claiming the DMI, and they concluded 
that the DMI "led to minimal improvement in 
quality of diabetes care at the population and 
patient level." 
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Access to Specialized Diabetes-care 
Providers 

A multidisciplinary team approach involving 
different health-care providers has been proven 
effective in delivering diabetes care. According to 
the Canadian Diabetes Association 2008 Clinical 
Practices Guidelines, "a team of health-care profes- 
sionals — including physicians, nurses, diabetes 
educators, pharmacists and other health-care 
experts who work together with the individual liv- 
ing with diabetes — is the recommended approach 
to achieve optimal care." The Diabetes Expert Panel 
appointed by the Ministry in 2006 also noted that 
the "interdisciplinary team is critical to the success- 
ful diabetes management." 

Our survey of the DEPs indicated that the 
levels of care and access to specialists varied from 
one DEP to another. Only 25% of the DEPs that 
responded to our survey said they were funded for 
specialists other than registered nurses and regis- 
tered dieticians. Although it is unlikely that there is 
sufficient funding to staff every DEP with an array 
of specialists, certain specialists are critical to dia- 
betes patient care. For instance, 60% of DEPs in our 
survey identified foot-care specialists as the most- 
needed professionals, but 40% said their patients 
had no ready access to these specialists. About half 
of the physicians who responded to our survey also 
identified "lack of timely access to foot-care special- 
ist" as one of the most common problems in caring 
for patients with diabetes, for whom foot infections 
are a major reason for hospitalization. According to 
a report published in April 2012 by the Institute for 
Clinical Evaluative Sciences, a leading independ- 
ent Canadian research organization, about 74 per 
10,000 Ontarians with diabetes have had lower 
extremity amputations that were mainly triggered 
by foot infections resulting from poor circulation 
and nerve damage caused by diabetes. 



Diabetes Regional Co-ordination Centres 

After Cabinet approved the Regional Coordination 
Centre (RCC) program in October 2009, the Min- 
istry established a diabetes RCC in each of Ontario's 
14 Local Health Integrated Networks (LHINs) to 
promote access to co-ordinated diabetes services 
regardless of where in Ontario people live. The 
Ministry and each LHIN selected a host agency (a 
hospital, community health centre or Community 
Care Access Centre) to manage the RCC. The RCCs 
do not deliver diabetes services directly, but work 
with their respective LHINs and health-care provid- 
ers to co-ordinate diabetes care in a region. 

The RCCs have been in place only since the 
2009/10 fiscal year, so it is not yet possible to assess 
their long-term impact. We did note that the RCCs 
have made some progress in improving regional 
co-ordination of diabetes services. However, the 
four RCCs we visited raised the following concerns, 
which they said must be addressed to enhance their 
effectiveness. Among them: 

• There is a need for clarity and standardization 
across regions regarding the interpretation 

of RCC deliverables, such as depth of support 
to service providers. However, the Ministry 
informed us that variation across the RCCs is 
needed to reflect local and regional needs. 

• Communication of RCC roles at the provincial 
level has been limited or ineffective. Although 
the RCCs have also made an effort to promote 
themselves, they acknowledged that many 
stakeholders, especially physicians, were still 
not aware of their role and mandate. The 
Ministry indicated that it provided formal 
communication about the RCCs in quarterly 
newsletters, at conferences and events, and 
through a refreshed diabetes website. 

• The RCCs have been leading regional diabetes 
planning and co-ordination by working closely 
with the DEPs, but the RCCs and the DEPs 
report to different branches of the Ministry, 
hindering efforts to evaluate their relative 
effectiveness. 
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• Despite monthly teleconferences and quar- 
terly face-to-face meetings, there has not been 
enough dialogue between the Ministry and 
the RCCs. For example, we noted cases where 
one branch of the Ministry did not consult 
and inform the RCCs when making changes 
at the DEPs. As well, although one branch 

of the Ministry asked the RCCs to do patient 
consultations, the RCCs found that another 
branch had also initiated a similar survey The 
RCCs also informed us that there is a need to 
increase the Ministry's participation in RCC 
stakeholder meetings to address the issues 
and barriers faced by the DEPs. 

• The RCCs have been developing their own 
tools and processes, such as patient surveys, 
performance indicators and guidelines for 
leading practices, concurrently, resulting in 
duplication of effort and inconsistencies in 
reporting and measurement of effectiveness. 
The Ministry informed us that since varia- 
tions exist across the RCCS, there is a need 
for flexibility to address local and regional 
circumstances. 

The Ministry acknowledged these issues and 
indicated that they are being addressed. In April 
2012, for example, the Ministry in partnership with 
the RCCs and Health Quality Ontario finalized a 
Measurement Framework that included measures 
to track and monitor regional performance. The 
Ministry advised us that it is working with the RCCs 
to identify common indicators for these measures in 
the 2012/13 fiscal year, and it has formally clarified 
expectations and respective roles of the RCCs and 
their host agencies. 



RECOMMENDATION 4 



To improve co-ordination among diabetes-care 
providers and access to specialized diabetes 
care, the Ministry of Health and Long-Term 
Care should: 

• take into account the demand for and avail- 
ability of diabetes services offered in com- 



munity health centres, hospitals and Family 
Health Teams when allocating diabetes fund- 
ing and other resources to avoid duplication 
or under-utilization of services; 

• evaluate the need for the Diabetes Man- 
agement Incentive, given the evidence 
indicating its lack of impact on encouraging 
physicians to provide continuous and co- 
ordinated diabetes management; and 

• monitor whether people have timely and 
equitable access to diabetes-care specialists 
in high demand, such as foot-care specialists, 
especially where there is evidence that a 
lack of timely treatment is likely to result in 
hospitalization. 



| MINISTRY RESPONSE 




The Ministry will undertake a review of its 
current operating framework to identify oppor- 
tunities to further reduce potential duplication 
and/or under-utilization. The current RCC 
program leverages information on the demand 
for, and availability of, diabetes services within 
local regions to drive effective and efficient co- 
ordination, integration and delivery of services. 

The Ministry continues to benefit from 
expert panels in strengthening primary care, 
which will be extended to include advice from 
the expert group on all physician incentives and 
their effectiveness, including the Diabetes Man- 
agement Incentive. 

The Ministry will, as appropriate, include 
equitable access to diabetes care specialists as 
part of its regular review and refinement of the 
Ontario Diabetes Strategy. 



BARIATRIC SURGERY 

About 90% of people with Type 2 diabetes are 
overweight or obese, according to the World Health 
Organization and the Canadian Diabetes Associa- 
tion. The International Diabetes Federation also 
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noted that the risk of developing Type 2 diabetes 
is 93 times higher for obese women and 42 times 
higher for obese men than for those at a healthy 
weight. 

Although bariatric surgery by itself does not 
"cure" obesity, it does result in substantial, sus- 
tained and long-term weight loss by removing 
part of the stomach and/or the small intestine. As 
one would expect, post-surgical follow-ups and 
ongoing diabetes management are critical to main- 
tain weight loss. Studies have consistently dem- 
onstrated that bariatric surgery is cost-effective 
for severely obese diabetes patients and stabilizes 
Type 2 diabetes in 60% to 80% of patients within a 
year or less. 

Access to and Cost of Bariatric Surgery 

Prior to the Strategy, most Ontarians requiring bari- 
atric surgery received it in the United States, with 
OHIP covering the costs. As part of the Strategy, 
the Ministry announced it would spend $75 mil- 
lion to increase Ontario's capacity to do bariatric 
surgeries. The Strategy called for the establishment 
of the Ontario Bariatric Network, consisting of six 
Regional Assessment and Treatment Centres and 
four Bariatric Centres of Excellence in hospitals 
across Ontario. Additional funding brought the 
total to about $108 million by the 2011/12 fiscal 
year. The number of in-province bariatric surgeries 
has increased from 245 in the 2007/08 fiscal year 
to 2,500 in 2011/12. 

Since February 2010, the Ministry has required 
Ontarians seeking bariatric surgery to be assessed 
at a Regional Assessment and Treatment Centre. 
Patients would be approved for out-of-country 
bariatric surgery only if their referring physicians 
applied for them and they met the regulatory 
requirements for out-of-country health services. 
The Ministry's goal was to have more bariatric 
surgeries performed in Ontario to save money and 
to ensure that adequate assessments and follow-ups 
are being done. 



Since the expansion of the capacity to perform 
them in-province, about 5,200 bariatric surgeries 
were performed in Ontario between 2009/10 and 
2011/12. Annually, the number of bariatric surger- 
ies done in Ontario increased by 180%, from about 
890 in the 2009/10 fiscal year to about 2,500 in 
2011/12. However, the total number of people get- 
ting the surgery actually dropped 14%, from about 
2,900 in the 2009/10 fiscal year to about 2,500 in 
2011/12, as illustrated in Figure 6. This is because 
significantly fewer out-of-country bariatric surger- 
ies are being approved. 

In February 2011, the Ministry announced 
that the expansion of bariatric services in Ontario 
would save the province about $45 million in out- 
of-country OHIP costs in the 2010/11 fiscal year. 
We noted that a more accurate estimate of savings 
would be about $35 million after including the 
cost of additional bariatric surgeries performed in 
Ontario. However, this figure does not reflect any 
potential costs of patients who could develop other 
health complications as a result of waiting longer 
for bariatric surgeries in Ontario, as indicated in the 
following sections. 



Figure 6: Number and Cost of Bariatric Surgeries, 
2005/06-2011/12 

Source of data: Ministry of Health and Long-Term Care 
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Capacity and Demand for Bariatric Surgery 

In May 2009, the Ministry estimated that 342,000 
Ontarians would be eligible for bariatric surgery 
and assumed that 2% (about 6,800) of those would 
proceed to the surgery by 2013. The Ministry also 
projected about 3,000 new referrals per year, based 
on data from 2005, then the most recently avail- 
able. Ministry data showed that in the 2011/12 
fiscal year, about 2,500 bariatric surgeries were 
performed in Ontario, as illustrated in Figure 6. 
This exceeded the Ministry's current target of 2,400 
but is not yet meeting the projected demand. 

Our research also indicated that the actual 
demand for bariatric surgery has significantly 
exceeded the forecasts. In the 2011/12 fiscal year 
alone, there were about 8,000 new referrals, or 
2.6 times more than the ministry forecast. We 
also noted that wait lists for bariatric surgery in 
Ontario suggest demand has far outpaced capacity. 
According to hospital data for the fiscal years 
between 2009/10 and 2011/12, there were about 
22,000 referrals. The Ministry informed us that 
about 70% of these referrals, or about 15,400 
cases, would proceed to surgery, but we noted that 
only about 5,200 bariatric surgeries were actually 
performed. The remaining referrals were either 
awaiting or undergoing pre-surgical assessment 
and preparation procedures. This gap will likely 
widen if demand continues to grow more quickly 
than capacity. We noted that the average overall 
wait time was 12 months if the surgery was done in 
Ontario, compared to six months if the surgery was 
approved to be done outside Ontario, mainly in the 
United States. 

The Ministry informed us that it has been dif- 
ficult to make accurate demand forecasts due to the 
elective nature of bariatric surgery. To address this 
challenge, the Ministry informed us that it has cre- 
ated a clinical registry to monitor quality and wait 
times. It also said that it will continue to allocate 
more resources to bariatric surgery. In the 2012/13 
fiscal year, for example, there will be a one-time 
funding allocation of about $48 million, and 
Ontario hospitals will be expected to perform 2,580 



bariatric surgeries and to treat 1,650 potential 
bariatric surgery patients in medical or behavioural 
programs as an alternative to surgery. 

Referral Process and Wait Times of 
Bariatric Surgery 

Alberta, British Columbia, Quebec and Saskatch- 
ewan also fund bariatric surgery in their provinces. 
Cross-jurisdictional comparison of the processes 
and wait times is not possible because policies and 
practices vary from one province to another. For 
instance, Ontarians seeking bariatric surgery follow 
a standard, centralized referral process to better 
ensure consistent assessment and prioritization of 
patients, as illustrated in Figure 7. The referral pro- 
cess in other provinces has not been centralized and 
each surgeon manages his or her own wait list. 

The Ministry collected wait-time data for 
bariatric surgery from the hospitals. However, we 
noted that two hospitals each reported identical 
wait times for every month from December 2010 
to November 2011, but neither was able to provide 
us with any documentation to support the wait 
times they reported to the Ministry. The Ministry 
indicated that it was likely due to the transition 
from a paper-based referral process to a centralized 
electronic system. Nevertheless, this made us ques- 
tion whether the wait-time data being collected 
by the Ministry was reliable, and accordingly, we 
measured wait time by reviewing patient files at 
the hospitals. 

As illustrated in Figure 7, the Ministry measured 
only the wait time from surgeon's approval to actual 
surgery, and did not include the period prior to 
surgeon's approval. According to the Ministry, this 
is the standard for tracking wait times for surgeries 
in Ontario. Using the Ministry's wait-time meas- 
ure, the average surgical wait time was only two 
months. However, our review found that patients 
waited an average of 12 months for surgery if the 
wait time was measured from the time of their 
physician's original referral to the actual surgery, as 
illustrated in Figure 7. 
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Figure 7: Referral Process for Bariatric Surgery in Ontario 

Prepared by the Office of trie Auditor General of Ontario 
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Our review of patient files and hospital data 
showed that the significant backlog for bariatric 
surgery was due not to a shortage of surgeons and/ 
or operating rooms but rather to the long time that 
patients waited between the original physician's 
referral to when the surgeon approved the surgery. 
Specifically: 

• Patients waited between two and seven 
months for a hospital orientation class, which 
bariatric surgery requires, possibly because 
classes were small and/or infrequent, or 
because of confusion among physicians about 
the referral process; physicians sometimes 
mistakenly sent referral forms to hospitals 
rather than to the central referral system. 
Hospital data showed that about 2,600 
patients (representing about 12% of total 
referrals from the 2009/10 to 2011/12 fiscal 
years) were waiting for orientation classes as 
of March 31, 2012. 

• Patients waited between five and eight 
months for specialist assessments. The waits 
depended on availability of hospital resources 
and level of patient needs, with some patients 
requiring more assessments before surgery. 
Hospital data showed that about 10,500 
patients (representing about 48% of total 



referrals from the 2009/10 to 2011/12 fiscal 
years) were waiting for or undergoing assess- 
ments as of March 31, 2012. 
While the Ministry's approach to reporting wait 
times for bariatric surgery is the same as that used 
for all surgeries in Ontario, we believe additional 
information regarding times that patients have to 
wait for orientation and assessment would provide 
the public with a better understanding of the 
referral process and what to expect. However, we 
also noted a McGill University Health Centre study 
published by the Canadian Medical Association in 
June 2009 which said that although most health 
authorities defined wait time as the period between 
a surgeon's approval and either hospital admission 
or actual surgery, "this same definition is inappro- 
priate for bariatric surgery because the enormous 
numbers of patients requesting this surgery would 
lead to a clinically unacceptable period from declar- 
ing a patient fit for surgery to the actual surgery. 
The realistic wait time for bariatric surgery must be 
defined as the overall wait time." 

Quality of Bariatric Services 

We found that the Ministry did not have adequate 
procedures in place to assess whether the referral 
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process for bariatric surgery operated as intended 
and whether patients received quality care. We 
noted, for example, that since the 2009/10 fiscal 
year, 29 bariatric surgeries were performed at 
hospitals not designated as a Bariatric Centre of 
Excellence. The Ontario Bariatric Network also 
informed us that it raised questions with the 
Ministry about the quality of services provided by 
non-designated hospitals. 

Since bariatric surgery has become more com- 
mon, there have been growing concerns about 
uneven quality across hospitals. In the United 
States, the American College of Surgeons Bariatric 
Surgery Center Network (College) and the Amer- 
ican Society for Metabolic and Bariatric Surgery 
(Society) accredit hospitals that perform bariatric 
surgery. In contrast, no professional medical bodies 
in Canada offer similar accreditation for bariatric 
surgery. Bariatric surgical sites in Ontario could 
also obtain accreditation from the United States, 
but of this province's eight sites, only two have 
obtained such an accreditation, both from the 
College. However, the Ministry informed us that it 
does not support U.S. -based accreditation, as the 
U.S. accreditation bodies require mandatory data 
submission, which would impose costs and admin- 
istrative burdens on Ontario's hospitals. The Min- 
istry also indicated that in the absence of Canadian 
accreditation, the Ontario Bariatric Network has set 
provincial standards and protocols for continuous 
quality improvement and monitoring of bariatric 
services in Ontario. 



RECOMMENDATION 5 



To ensure that people receive adequate, timely 
and quality bariatric surgical services across the 
province, the Ministry of Health and Long-Term 
Care should: 

• review trends of demand and capacity for 
bariatric surgery to identify gaps and needs, 
especially on a regional basis; 

• consider providing the public with informa- 
tion on the average elapsed time between 



a physician's referral and completion of the 
required pre-surgery assessments; and 
• periodically monitor surgical outcomes to 
determine whether hospitals offering this 
surgery need to go through an accreditation 
process as hospitals in the United States do. 

MINISTRY RESPONSI 

The Ontario Bariatric Network (OBN) reviews 
the trends of demand and capacity on a regular 
basis and provides updates and recommenda- 
tions to the Ministry. In 2009/10, the Ministry 
established the Bariatric Registry to provide reli- 
able data and to track both care-path timelines 
and performance measures, and will continue 
to refine the use of this data. In partnership with 
hospitals and the OBN, adjustments are made 
on a continuous basis, such as transferring 
patients between centres to reduce wait times, 
modifying referral zones and allocating addi- 
tional resources, where appropriate, in order to 
increase surgical volumes. 

Patients are advised of the referral process 
(including full wait-time information) and what 
they can expect during the information sessions 
that all patients are required to attend. The Min- 
istry will consider options for advising the pub- 
lic on full wait times from the original date of 
referral. The Ministry currently publishes wait 
times for bariatric surgery through the Ontario 
Wait Times Information System. Between April 
2012 and June 2012, the wait time for bariatric 
surgery once a surgeon had approved it was 85 
days, which is well within the general surgery 
target of 182 days. 

The Ministry will continue to regularly 
monitor surgical outcomes of this program, 
including reviewing quality benchmarks and 
monitoring improvement plans as required for 
any bariatric centre. 



Chapter 3 
Section 




Ministry of the Environment 

Drive Clean Program 



Background 




Smog is a form of air pollution that poses a serious 
health threat to Ontarians. According to the Min- 
istry of the Environment (Ministry), approximately 
half of Ontario's smog comes from pollutants that 
originate in the United States and are transported 
here by winds. The other half, however, comes from 
domestic sources, including utilities (for example, 
power plants), industries (for example, metal 
smelters and petroleum refineries), on-road motor 
vehicles and other forms of transportation (for 
example, trains and aircraft). Vehicles also contrib- 
ute to greenhouse gas emissions (such as carbon 
dioxide) and toxic contaminants (such as carbon 
monoxide and benzene), which also adversely 
affect air quality. For more than a decade, the 
Ministry has been implementing a number of initia- 
tives aimed at helping to reduce smog. One of these 
initiatives has been the Drive Clean program. 

Drive Clean, which was introduced in 1999, is 
Ontario's mandatory vehicle emissions inspection 
and maintenance program. The purpose of the pro- 
gram is to identify high-pollution-emitting vehicles 
with missing or malfunctioning emission controls 
and require the owners of such vehicles to have 
them repaired, with the ultimate goal of reducing 
on-road emissions. The program has two compon- 



ents — one for light-duty vehicles (passenger cars 
and sport utility vehicles) and one for heavy-duty 
vehicles (buses and trucks) . The program cur- 
rently tests vehicles once they are seven years old, 
or those older than one year if ownership is to be 
transferred. Light-duty vehicles that were built 
before 1988 are exempt from the program, but, 
otherwise, all vehicles must pass an emissions test 
for the owner to renew the registration or transfer 
ownership. When a vehicle passes its emissions 
test, the testing facility issues a uniquely numbered 
emissions test certificate, which the vehicle owner 
must take to a ServiceOntario kiosk or office (that 
is, a vehicle-licensing office) when applying to 
renew their licence plates or to transfer ownership. 

As of December 2011, approximately 7.6 million 
light-duty vehicles and almost 300,000 heavy-duty 
vehicles were registered in Ontario. About 90% 
of these vehicles are registered in the geographic 
area covered by the program. Annually, more 
than 2.5 million light-duty vehicles and more than 
100,000 heavy-duty vehicles are subject to the 
Drive Clean test. 

More than 30 jurisdictions in North America 
have a vehicle inspection and maintenance pro- 
gram for emissions. In Canada, British Columbia 
has been testing light-duty vehicles since 1992, 
and recently announced that it will phase out that 
program by the end of 2014 and start testing heavy- 
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duty diesel vehicles only. Quebec, on the other 
hand, introduced a bill in December 2011 to begin 
emissions testing for light-duty vehicles by the end 
of 2013. 

Ontario's Drive Clean program operates under 
the authority of regulations made under the 
Environmental Protection Act and the Highway Traf- 
fic Act. The program is administered jointly by the 
Ministry of the Environment, the Ministry of Trans- 
portation and ServiceOntario, with the Ministry of 
the Environment being ultimately accountable for 
its performance. 

The Ministry of the Environment's Drive Clean 
Office, with a staff of approximately 25 people, 
develops regulations, sets standards, establishes 
policies and procedures, and contracts with service 
providers to deliver various aspects of the program. 
Emissions tests and/or repairs are performed at 
more than 2,000 Drive Clean facilities, which are 
private auto shops accredited by the Ministry. All 
testing facilities are electronically linked to the 
Ministry's Drive Clean database, which maintains a 
record of all tests and any related repairs made. 

The Ministry has contracted with a private- 
sector service provider to administer all operational 
Drive Clean program activities. This involves 
supplying and servicing emissions testing equip- 
ment, training inspectors and repair technicians, 
ensuring quality control practices at Drive Clean 
facilities, operating a call centre to provide tech- 
nical support to Drive Clean facilities and handle 
public comments, and developing and operating an 
information system to support all of the program's 
functions, including linking to the Ministry of 
Transportation's licensing system. 

Vehicle owners pay a fee to the Drive Clean 
facility that conducts their emissions test. A portion 
of this fee is remitted to the Ministry as revenue. 
In the 2011/12 fiscal year, the Ministry collected 
$30 million in test revenue and spent approxi- 
mately $19 million to deliver the Drive Clean 
program, of which $12 million was paid to the 
private-sector service provider. 



In 2012, the Ontario government commissioned a 
review on reforming public services in the province. 
The resulting report, commonly referred to as the 
Drummond Report, recommended that the govern- 
ment consider delivering the Drive Clean program 
via a "delegated administrative authority" model. A 
delegated administrative authority is a private, not- 
for-profit corporation that administers legislation on 
behalf of the government under an accountability 
and governance agreement. This entity would 
assume responsibility for all aspects of the program's 
day-to-day decision-making and service delivery that 
are now being handled by the Drive Clean Office and 
its private-sector service provider. 



Audit Objective and Scope 



The objective of our audit was to assess whether 
the Ministry has adequate systems and procedures 
in place to ensure compliance with legislation 
and regulations related to Drive Clean, and to 
determine and report on whether the program is 
effective in reducing vehicle emissions and thereby 
contributing to improved air quality. Senior Min- 
istry management reviewed and agreed to our audit 
objective and associated audit criteria. 

Our audit work was conducted primarily at the 
Ministry's Drive Clean Office and at the private- 
sector service provider. In conducting our audit, 
we interviewed appropriate ministry and service- 
provider staff, reviewed relevant documents, ana- 
lyzed information and reviewed specific controls of 
new testing equipment that is slated for rollout in 
January 2013. In addition, we employed a number 
of computer-assisted audit techniques to analyze 
two sets of data: the results of emissions tests and 
the records on repair costs. 

We met with the Canadian Vehicle Manufactur- 
ers' Association and the Environmental Commis- 
sioner of Ontario to obtain their perspectives on 
the Drive Clean program. To gain insight on how 
similar programs operate in other jurisdictions, we 
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reviewed studies and reports on vehicle inspection 
and maintenance programs elsewhere in Canada 
and in the United States. Because the Ministry's 
internal audit service team had recently performed 
audits on Drive Clean revenue, we took its work 
into consideration in determining the scope and 
extent of our work in this area. 



Summary 




The Drive Clean program has implemented effect- 
ive procedures to ensure that vehicles that should 
be tested are getting tested, and that vehicles 
whose emissions systems have deteriorated to the 
point where their emissions exceed the province's 
limits are being identified for repair. The Ministry 
has made some headway in refining the program's 
features — for example, increasing the age at which 
vehicles must begin emissions testing, extending 
testing to include vehicles that in 2012 were as old 
as 24 years, changing the vehicle emissions testing 
method to the on-board diagnostic testing method 
that is now used in all other North American juris- 
dictions with similar programs, and consolidating 
six alternative service delivery contracts into one 
at an expected annualized cost savings of 40%. 
The Ministry, together with the Ministry of Trans- 
portation, has also put in place procedures aimed 
at preventing the use of duplicate certificates, a 
fraudulent practice that was identified as a signifi- 
cant problem in our 2004 audit on air quality. 

On-road vehicle emissions have declined so 
significantly from 1998 to 2010 that they are no 
longer among the major domestic contributors of 
smog in Ontario. However, ministry emissions esti- 
mates show that more than 75% of the reduction in 
vehicle emissions since the program's inception is 
actually due to factors other than the Drive Clean 
program, such as tighter manufacturing standards 
on emission-control technologies, federal require- 
ments for cleaner fuels and ongoing retirement 
of old vehicles. For emissions not eliminated by 



these factors, the Ministry further estimated that, 
since 2007, the Drive Clean Program has been 
responsible for reducing smog-causing vehicle 
emissions by about 36% annually. Initiatives in 
other sectors (for example, changes in industrial 
processes) have also contributed to the reduction in 
smog-causing emissions. 

It is therefore critical that, on a go-forward 
basis, policymakers have relevant and up-to-date 
information on the actual impact of the Drive Clean 
program in reducing smog compared to the impact 
of other smog-reducing initiatives. 

There are a number of issues that we noted dur- 
ing our audit: 

• The worst polluting vehicles either are exempt 
from emissions testing or will be tested using 
a less stringent method. The program's light- 
duty component does not require vehicles 
built before 1988 to be tested, even though 
they would likely have about a 30% failure 
rate. In addition, the on-board diagnostic 
(OBD) testing method that is slated to begin 
January I, 2013, cannot be used to test 
vehicles built before 1998 because these 
vehicles were built without OBD systems. As 

a result, vehicles built from 1988 through 
1997, which experienced a failure rate from 
11% to 31% in 2010 when tested with a 
dynamometer, will be tested using the two- 
speed idle method only — a method that uses 
less stringent emissions limits than either the 
dynamometer or the OBD testing method. 

• Since 2002, all gasoline-powered vehicles that 
meet the program's age criteria and that are 
located in the Windsor-Quebec City corridor 
are required to have emissions tests every two 
years. There are 10 large municipalities in this 
geographical area that account for about two- 
thirds of all vehicles in Ontario. These muni- 
cipalities generally also have a higher than 
average number of smog days. However, the 
Ministry has never formally assessed whether 
there would be any significant impact on the 
environment if vehicles not located in these 10 
municipalities were not required to be tested. 
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• Because vehicle owners are not required to 
incur any repair costs if the repair estimate 
exceeds $450, about 18,000 vehicles avoided 
being fully repaired in 2011. In fact, the 
average amount paid for repairs in 2011 

by the owners of vehicles that were given a 
conditional pass was only $255. Furthermore, 
in 2010 the most commonly diagnosed cause 
of excessive emissions — problems with the 
catalytic converter — was actually repaired in 
only one-third of cases. Without full repairs, 
a vehicle's emission control system will 
continue to malfunction, and emissions will 
fluctuate. We noted that, for vehicles that had 
partial repairs done to their emission systems 
in 2011, the emission readings after the repair 
were actually worse for all pollutants in 25% 
of the vehicles and for at least one of the pol- 
lutants measured in 50% of the vehicles. 

• To uncover unscrupulous practices at Drive 
Clean facilities, the Ministry has, until 
recently, been diligent in requiring its service 
provider to conduct upwards of 1,400 covert 
audits annually. In recent years, these and 
other audit efforts have been effective in iden- 
tifying approximately 3,000 non-compliance 
issues annually. However, with the planned 
introduction of a new compliance program to 
coincide with the new OBD testing method 

in 2013, the Ministry reduced the number of 
covert audits in 2012 to a very small fraction 
of what the Ministry previously required 
the service provider to conduct. While a 
significant decrease may well be justified 
once the new compliance program is in place, 
we believe the Ministry should reconsider 
whether such a dramatic reduction is cur- 
rently warranted, especially given that there 
is a deterrent effect when Drive Clean facility 
operators know that covert or "secret shop- 
per" audits are being done. 

• Although one of the program's key goals is 
maintaining a high degree of public accept- 



ance, the Ministry has not established 
performance targets or attempted to measure 
whether or not this goal has been achieved in 
over a decade. The only survey undertaken to 
measure public support for the Drive Clean 
program was conducted 12 years ago. 

OVERALL MINISTRY RESP0NSI 

The Ministry appreciates the Auditor General's 
observations and recommendations regarding 
the Drive Clean program and will continue to 
take actions to improve the program. 

Poor air quality is a public health concern, 
particularly to children, the elderly and people 
who have respiratory challenges. Ontario 
protects air quality through its comprehensive 
approach, which includes a combination of 
regulations, compliance and enforcement 
activities, monitoring, and Drive Clean. The 
approach tackles emissions from the electricity 
sector, commercial and industrial facilities, 
vehicles, and cross-border sources. Our regula- 
tions and targeted programs and partnerships 
with other jurisdictions have all helped to 
reduce air pollutants. To put the Drive Clean 
program's mandate into context, on-road 
vehicles account for 27% of all emissions of 
nitrogen oxides in the province and 13% of all 
emissions of volatile organic compounds. Since 
2007, the program has consistently reduced 
vehicle emissions that cause smog and poor air 
quality by approximately 36% a year. 

The Ministry notes the Auditor General's 
observation that on-road emissions have been 
continuously declining. Still, parts of Ontario 
experience elevated smog levels. Since the 
beginning of the program, Drive Clean has 
reduced smog-causing pollutants by 335,000 
tonnes; carbon monoxide by about 3.18 million 
tonnes; and carbon dioxide, a key greenhouse 
gas, by more than 296,000 tonnes. In 2003, 
when operating at their peak, coal-fired gen- 
erating plants emitted over 58,000 tonnes of 
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smog-causing pollutants. In 2010 alone, Drive 
Clean prevented 34,600 tonnes of smog-causing 
pollutants. 

A review of the program's costs, benefits 
and effectiveness was an important driver in 
the Ministry's 2010 decision to modernize the 
Drive Clean program. The Ministry consolidated 
multiple service contracts into one contract to 
improve efficiency and effectiveness of delivery. 
The new vehicle emissions testing technol- 
ogy will allow problems to be sourced faster 
and more effectively. We anticipate further 
decreases in smog-causing pollutants with our 
new modernized program, and once the tech- 
nology is implemented in 2013, we will continue 
to monitor the program's effectiveness. 

The Ministry is taking action to address the 
concerns raised by the Auditor General. The 
new approach to testing vehicle emissions is 
used in similar programs across North America. 
It is expected to: 

• provide better information to vehicle owners 
about needed repairs; 

• reduce the number of conditional passes; 

• ensure high-quality customer service thanks 
to online real-time quality control and assur- 
ance tools; and 

• continue to improve Ontario's air quality by 
reducing emissions an additional 20%. 



PROGRAM EFFECTIVENESS 

Overall Change in Ontario's Air Quality 

Smog is a form of air pollution that is composed 
primarily of ground-level ozone and fine particulate 
matter. Ground-level ozone is created when nitro- 
gen oxides (NOx) and volatile organic compounds 
(VOCs) react together in the presence of sunlight. 
Fine particulate matter (PM2.5) is microscopic solid 
particles and liquid droplets in the air. 

Wind-blown pollution from the Midwestern 
United States was and remains the largest source 
of smog-causing pollutants found in Ontario. How- 
ever, as Figure 1 shows, between 1998 and 2010, 
Ontario-based emissions from smog-causing pol- 
lutants (NOx, VOCs, and PM2.5) and carbon mon- 
oxide have decreased by about 30% to 40%. The 
Ministry's report on air quality for 2010 credited the 
decrease in emissions to a combination of federal 
and provincial air quality initiatives: 

• the phase-out of coal-fired generating stations 
(between 2005 and 2014); 

• emission trading regulations (effective 2001 
for the electricity sector and 2005 for the 
industrial sector); 

• emission controls at Ontario smelters (effect- 
ive 2005); 

• Drive Clean emissions testing (beginning 1999); 



Figure 1: Changes in Province-wide Domestic Emissions and Contribution by Road Vehicles to Those Changes, 
1998-2010 

Source of data: Ministry of the Environment 
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• federal vehicle manufacturing standards for 
emissions (phased in from 2004 through 
2010); and 

• the federal requirement for lower sulphur con- 
tent in transportation fuels (effective 2005 for 
gasoline fuel and effective 2006 for diesel fuel). 

Just before the Drive Clean program started, 
vehicles were the number one domestic source 
of emissions for carbon monoxide and nitrogen 
oxides, and the number two domestic source of 
emissions for volatile organic compounds. As of 
2010, vehicles were still the primary domestic 
source of carbon monoxide emissions but were no 
longer the primary source of domestic emissions 
for any of the smog-causing pollutants listed in 
Figure 1. For each year from 1998 through 2010, 
the vehicle sector experienced either the largest or 
the second-largest decrease in emissions for each 
of these pollutants, and total vehicle emissions 
decreased more than 50% in that time. Other 
sources — such as non-road transportation (which 
includes air, rail and marine transportation) — 
demonstrated the least reduction in smog-causing 
emissions, as shown in Figures 2 and 3. 

We also looked at ambient air quality measures, 
which monitor the concentration of contaminants 
in the air for a select period of time. Unlike emis- 



sion levels, ambient air quality measures provided 
mixed results. That is, for calendar years 2008 
through 2010, Ontario did not exceed the Canada- 
wide standard for PM2.5, but did exceed the 
Canada- wide standard for ozone in each of those 
years, although ozone concentrations have been 
decreasing for some time. 

When smog levels are (or are expected to 
become) elevated for several hours, the Ministry 
issues a smog advisory. The total hours spent under 
a smog advisory during a particular time period can 
be expressed as "smog days": every 24 hours under 
a smog advisory equals one smog day. We analyzed 
the number of smog days experienced province- 
wide each year since the Drive Clean program's 
inception and noted that smog days varied con- 
siderably from year to year but have been generally 
trending down. 

Impact of the Drive Clean Program 

To determine the effectiveness of the Drive Clean 
program, the Ministry has engaged an external 
consultant to calculate emissions reductions. Gen- 
erally, emissions reductions are calculated based on 
the difference between a vehicle's actual emissions 
readings before and after repairs are conducted 



Figure 2: Nitrogen Oxide Emissions, by Domestic 
Emission Source, 1998-2010 (Kilotonnes) 

Source of data: Ministry of the Environment 
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Figure 3: Volatile Organic Compound Emissions, by 
Domestic Emission Source, 1998-2010 (Kilotonnes) 

Source of data: Ministry of trie Environment 
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after the vehicle fails its initial Drive Clean test, 
factoring in the distance typically travelled by the 
repaired vehicle in the year. The total is then fur- 
ther extrapolated to account for vehicles that were 
not actually tested in a given year (because vehicles 
are tested only every two years) and to estimate 
kilometres travelled by vehicles tested for the first 
time (for which the system therefore contains no 
previous odometer reading) . 

We reviewed the consultant's 2010 Emissions 
Benefit Analysis reports for the three vehicle types 
covered by the program, and noted that the pro- 
gram is believed to have had the following impact 
on emissions: 

• For heavy-duty diesel vehicles, particulate 
matter emissions (the only pollutant that 
can be measured using the available testing 
technology for such vehicles) have been 
reduced by an average of 250 tonnes each 
year throughout the program's existence. 

• For heavy-duty non-diesel vehicles, the 
consultant concluded that the impact of the 
program was negligible because only a few 
such vehicles are on Ontario's roads. 

• For light-duty vehicles, emissions were reduced 
by steadily increasing amounts from 1999 
through 2007. Since then, although emissions 
have still been reduced annually, the amount 
by which they've been reduced has been 
declining, as shown in Figure 4. According to 
the consultant's estimates, the Drive Clean 
program has been responsible for reducing 
smog-causing vehicle emissions by about 36% 
a year since 2007. The consultant noted a simi- 
lar trend for carbon monoxide emissions. 

However, as Figure 5 indicates, the consultant 
estimated that had the Drive Clean program not 
existed, smog-causing emissions from light-duty 
vehicles would still have decreased by 54% from 
1999 to 2010 because of other factors — such as the 
retirement of older vehicles, the introduction of 
vehicles with cleaner emissions control technolo- 
gies, and fuel improvements. The vehicle repairs 
required as a result of the Drive Clean program 



Figure 4: Reduction in Emissions of Smog-causing 
Pollutants (Nitrogen Oxide and Hydrocarbons) 
Attributable to the Drive Clean Program for Light-duty 
Vehicles, 1999-2010 (Tonnes) 

Source of data: Ministry of the Environment 
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Figure 5: Vehicle Emissions of Smog-causing Pollutants 
(Nitrogen Oxide and Hydrocarbons) with and without 
the Drive Clean Program, 1999-2010 (Tonnes) 

Source of data: Ministry of the Environment 
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were estimated to have reduced emissions by a 
further 16%, for a total reduction in emissions of 
70%. In other words, more than 75% of the total 
emissions reductions for light-duty vehicles from 
1999 to 2010 can be attributed to factors other than 
the Drive Clean program. 
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The Ministry indicated to us that it believes the 
Drive Clean program has also led to changes in 
vehicle owners' behaviours that cannot be quanti- 
fied: for example, vehicle owners make an increased 
effort to maintain their vehicles in order to be able 
to pass the Drive Clean test, and vehicle owners may 
decide to replace vehicles sooner than they other- 
wise might have if the program did not exist. 

Given the declining impact of the program on 
air quality, we reviewed practices in other jurisdic- 
tions and noted that British Columbia announced 
in May 2012 that it will terminate its program for 
light-duty vehicles by the end of 2014, because it 
determined that vehicles were no longer one of the 
primary contributors of pollutants in that province. 
Five U.S. jurisdictions have also ended their pro- 
grams, and one other has announced plans to do 
so in 2012, either because emissions have returned 
to acceptable levels or because other methods 
are expected to have a more significant impact on 
reducing air pollutants. 

Emissions Test Results 

Over the Drive Clean program's life, initial pass 
rates — that is, the rates at which vehicles of a par- 
ticular type passed their emissions tests on the first 
try — have improved. As shown in Figure 6, initial 
pass rates are currently 90% or more for all vehicle 
types tested. For light-duty vehicles, the initial pass 
rate increased from 84% in 1999 to 95% in 2010, 
and has exceeded 90% every year since 2004. 
Heavy-duty non-diesel vehicles have shown the 
greatest improvement, with an initial pass rate that 
increased from 72% in 1999 to 90% in 2010. Heavy- 
duty diesel vehicles have continuously had the high- 
est initial pass rates throughout the program's life. 
Since 2001, their initial pass rate has exceeded 95%. 

Reporting Other Performance Achievements 

The Drive Clean program has four key goals: 
• reducing vehicle-related emissions of smog- 
causing pollutants; 



• attaining a high degree of public acceptance; 

• achieving revenue neutrality over the pro- 
gram's lifespan, with full-cost recovery via test 
fees; and 

• maintaining business integrity (that is, zero 
tolerance for fraud) . 

We found that the Ministry does not have 
quantifiable targets and measures for most of these 
goals. The Ministry has set a published target for 
only one performance measure — emission reduc- 
tions. In Ontario's Anti-Smog Action Plan for 2000, 
the Ministry stated that by the fiscal year 2005/06, 
the Drive Clean program was expected to achieve 
a 22% reduction in vehicle emissions, for nitrogen 
oxides and volatile organic compounds (VOCs) 
combined. But the Ministry has not reported 
against this target either in its annual Air Quality 
Report or in the annual Emissions Benefit Analysis 
reports prepared by external consultants. In fact, 
the Emissions Benefits Analysis reports do not 
report reductions of VOCs but rather reductions of 
hydrocarbons, which are a component of VOCs. As 
previously noted, the reported cumulative emis- 
sions reduction of nitrogen oxides and hydrocar- 
bons attributable to the Drive Clean program from 
its inception to the end of 2010 was 16%. 

Figure 6: Initial Pass Rates for Light-duty Vehicles, 
Heavy-duty Non-diesel Vehicles and Heavy-duty Diesel 
Vehicles, 1999-2010 (%) 

Source of data: Ministry of the Environment 
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We noted that the Ministry had also set internal 
targets that appeared to be easily attainable. To 
illustrate, the Ministry establishes an annual inter- 
nal target for emissions reductions for non-diesel- 
powered vehicles. We were informed that this 
target was established by projection based on previ- 
ous years' actual results, yet the new target was 
generally lower than the actual emissions reduc- 
tions achieved in the previous year. For example, 
since emissions reductions from 2007 through 

2009 declined by 8% annually, we expected the 

2010 emissions reduction target to be about 32,000 
tonnes, yet the Ministry set a 2010 target of only 
28,500 tonnes. 

With regard to the program's other goals, we 
found the following: 

• Although the Ministry informed us that it had 
conducted a public survey in 2000 to measure 
the level of public acceptance for the program, 
no additional public surveys had been con- 
ducted in the ensuing 12 years. 

• For the goal of business integrity, the Ministry 
discloses on its website a list of individuals 
and Drive Clean facilities convicted of fraud- 
related offences. It also discloses a list of Drive 
Clean facilities that have been terminated 

or suspended in the last three years. The 
Ministry could build on these good disclosure 
measures by reporting on the number or 
percentage of facilities that are in compliance 
with contract requirements and standard 
operating procedures. 
Since 2005, the Ministry has been publishing 
reports on emissions reductions resulting from 
the Drive Clean program. However, we found 
that the Ministry did not report on a timely basis. 
For example, consultants' reports on Drive Clean 
emissions reductions for 2009 and 2010 were not 
publicly released on the ministry website until June 
2012, and Ontario's latest Air Quality Report for 
2010 was not publicly released until April 2012. 



RECOMMENDATION 1 



To ensure that policy-makers are provided with 
current and relevant information, the Ministry 
of the Environment should formally evaluate the 
extent to which the Drive Clean program con- 
tinues to be an effective initiative in reducing 
smog relative to the cost and impact of any other 
initiatives for reducing smog and improving 
overall air quality. In addition, the Ministry 
should periodically evaluate its progress against 
all stated program goals and report the results 
of its assessments publicly on a timely basis. 



| MINISTRY RESPONSE 




The Ministry agrees that it is critical for policy- 
makers to have the best decision-making infor- 
mation available. 

As new data from the modernized program 
services and emission tests becomes available, 
the Ministry will continue to evaluate the 
progress and success of the Drive Clean pro- 
gram relative to its goals and other initiatives 
in Ontario. 



VEHICLES SUBJECT TO TESTING 

Vehicle Age 

For both light-duty and heavy-duty vehicles, the 
Drive Clean program exempts from testing any 
vehicle built in the last seven years. Vehicles exempt 
from testing and other program details are shown 
in Figure 7. Over the program's life, the Ministry has 
several times increased the age at which vehicles 
must begin being tested: when the program started, 
that age was 3; in 2006, it was changed to age 5; 
and in September 2011, it was changed again, to 
age 7. In comparison, British Columbia also exempts 
vehicles built in the last seven years; Quebec plans 
to exempt vehicles built in the last eight years; and 
two U.S. jurisdictions, including California, exempt 
vehicles built in the last six years. In October 2012, 
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Figure 7: Drive Clean Program Details 

Prepared by the Office of the Auditor General of Ontario 




Light-duty vehicles 




Program Area 


Southern Ontario from Windsor to Ottawa 


Diesel: all of Ontario 
Non-diesel: Southern Ontario 


Types of Vehicles 
Tested 


Vehicles weighing 4,500 kg or less (e.g., passenger cars, 
vans, light trucks and sport utility vehicles), beginning 
with 1988 models that are registered in the program area 


Vehicles weighing more than 4,500 kg (e.g., 
large trucks and buses) that are registered 
in the program area 


Vehicles Exempt 
from Testing 


Vehicles manufactured before 1988, hybrid vehicles, 
designated "historic" vehicles, light-duty commercial farm 
vehicles, kit cars and motorcycles 


Vehicles that are designated as "historic" 
according to the Highway Traffic Act 


Testing Frequency 


• Every two years, beginning when vehicle is 7 years old, 
to renew registration. Odd model years are tested in 
even calendar years. 

• When transferring ownership of a used vehicle to 
someone other than a family member or the lessee, 
if the vehicle has not been tested in the previous 12 
months. 


• Every year, beginning when the vehicle is 
7 years old, to renew registration. 

• Diesel vehicles may skip a year of testing 
if their opacity reading was 20% or less 
in their most recent test. 

• When transferring ownership of a used 
vehicle older than the current year, if 
the vehicle has not been tested in the 
previous 12 months. 


Testing Methods 


(Effective until December 31, 2012) 

• Accelerated Simulation Mode - for most vehicles. 
Testing is conducted on a dynamometer (i.e., vehicle 
treadmill) 

• Two-speed idle test - for vehicles that cannot be safely 
tested on a dynamometer 

• Visible smoke test - for diesel vehicles only 
(Effective January 1, 2013) 

• On-board diagnostic test - for post-1997 models 

• Two-speed idle test - for pre-1998 models 

• Visible smoke test - for diesel vehicles only 


Diesel: Opacity smoke test 
Non-diesel: Two-speed idle test 


Emissions Measured 


Carbon monoxide, hydrocarbons, nitrogen oxide 


Diesel: Particulate matter 
Non-diesel: Carbon monoxide and 
hydrocarbons 


Possible Test Results 


Pass / Conditional Pass / Fail 


Pass / Fail 


Conditional Passes 


Allowed only where the purpose for testing is to renew 
vehicle registration, not for ownership transfer. Estimates 
and any actual repairs must be done by a certified repair 
technician at an accredited Drive Clean facility (DCF). 


Not allowed. All vehicles must be repaired 
to the point where they pass an emissions 
test. 


Repair Cost Limit 


$450 


None 


Who Can Perform 
Repairs 


Anyone and any auto shop, but only those repairs 
performed by a certified repair technician at a DCF are 
eligible for a conditional pass 


Anyone 


Program 
Infrastructure 


Program is delivered through a decentralized network of 
test-only, test-and-repair, and repair-only DCF locations 


Program is delivered through a decentralized 
network of test-only DCFs, some of which 
provide mobile testing for diesel vehicles 


Cost of Test to 
Vehicle Owner 


Up to $35 per test; re-tests cost $17.50, if done at the 
same DCF as the initial test 


Market rate 


Ministry Revenue 


$11.67 of fee charged by DCF for a pass or conditional 
pass 


$15 for each passed test 
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Ontario's Environmental Commissioner supported 
the new seven-year exemption limit in his latest 
annual report. 

Any vehicle older than a year whose ownership 
is being transferred to a third party is required to 
undergo an emissions test, as a consumer protec- 
tion feature for the prospective new owner. 

As of December 2011, half of all light-duty 
vehicles registered in Ontario had been built in the 
last seven years. For those in this category that had 
an emissions test in 2010 (likely due to an owner- 
ship transfer), the initial failure rate was 1% or less, 
as shown in Figure 8. Since these newer vehicles 
were built to stricter manufacturing standards for 
emissions, it is reasonable to assume that with 
proper maintenance, they will operate more cleanly 
as they age compared to previous model years. That 
is, when a newer vehicle is 10 years old, it will emit 
lower levels of pollutants than an older vehicle did 
when it was 10 years old. In contrast, initial failure 
rates for vehicles built in 1997 or earlier ranged 
from 11% to 31% in 2010. The Ministry has not 
compiled pass/fail rates by model year for heavy- 
duty vehicles. 

Prior to 2007, the program also exempted light- 
duty vehicles that were more than 20 years old. As 
of 2007, the program instead exempts vehicles built 
before 1988 (that is, at least 24 years old in 2012). 
This is because vehicles built before 1988 were not 
required by the federal government to have built-in 
emissions-reduction controls (that is, a catalytic 
converter, which converts toxic gases from a 
vehicle's exhaust system into less toxic substances) . 
As a result, vehicles that are potentially among the 
worst polluters could legally be on the road. In com- 
parison, about 80% of other North American juris- 
dictions, including British Columbia and California, 
require emissions testing for vehicles older than 24 
years. As of December 2011, more than 100,000 
pre-1988 vehicles were registered in Ontario. 
Because older vehicles have less stringent emis- 
sions limits than newer vehicles, an older vehicle 
that fails an emissions test will pollute more, per 
kilometre driven, than a newer vehicle that fails. 



Figure 8: Initial Failure Rates by Model Year for Light- 
duty Vehicles Tested in 2010 (%) 

Source of data: Ministry of the Environment 
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The Ministry noted that this situation is mitigated 
by the fact that, according to odometer data col- 
lected by the Ministry of Transportation when a 
vehicle is registered or re-registered, older vehicles 
are not driven as often or as far as newer vehicles. 
For example, in 2010, 2007 light-duty passenger 
vehicles were driven on average three times farther 
than 1987 light-duty passenger vehicles (17,000 
km versus 5,000 km, respectively). However, we 
noted a similar phenomenon with heavy-duty diesel 
vehicles — that is, in 2010, 2007 heavy-duty diesel 
vehicles were driven on average six times farther 
than 1987 heavy-duty diesel vehicles (52,000 km 
versus 9,000 km, respectively) — yet the Ministry 
does not exempt pre-1988 heavy-duty diesel 
vehicles from emissions testing. 

While the Ministry analyzed the impact of 
increasing the exemption age for new vehicles from 
five years to seven years and determined that the 
change will have little impact on total emissions, no 
analysis was done on the impact of excluding older 
vehicles from the program. 

Geographical Area Covered by the Program 

All heavy-duty diesel vehicles registered anywhere 
in Ontario that meet the age criteria (specified 
in the previous section) are required to be tested 
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under the Drive Clean program. In contrast, only 
those light-duty vehicles and heavy-duty non-diesel 
vehicles registered in a defined geographical area 
called the "program area" are subject to testing, 
assuming they also meet the age criteria. The 
program area can generally be considered to be the 
Windsor-Quebec City corridor. This corridor was 
selected as the program area because it had been 
identified in the 1990s by the Canadian Council 
of Ministers of the Environment as one of three 
problem areas in Canada that experienced higher- 
than-acceptable levels of smog. About 90% of the 
province's passenger vehicles are located in this 
part of the province. 

The program area for light-duty vehicles and 
heavy-duty non-diesel vehicles has not been 
reviewed since 2002, when it was last expanded. 
Although the current program area contains more 
than 30 municipalities, 10 municipalities, or about 
one-third, account for two-thirds of the province's 
passenger vehicles and population. In addition 
to having the highest vehicle density, all but one 
of these municipalities also experienced more 
smog days than the provincial average every year 
from 2005 through 2010. The Ministry has never 
formally assessed whether excluding those vehicles 
not located in these 10 municipalities from the 
required biennial testing could be done with little 
or no adverse effect on the environment. 

Farm Vehicles 

According to O. Regulation 628 of the Highway 
Traffic Act, vehicles registered to farmers are exempt 
from the Drive Clean program. The Ministry 
informed us that this exemption was put in place 
to ease the economic burden of emissions testing 
and the required repairs on farmers. However, we 
noted that when these vehicles are registered or re- 
registered, no verification is required to ensure that 
their vehicle owner is indeed a farmer. In essence, 
applicants merely have to tick a box on a form iden- 
tifying themselves as such. We noted that from 1998 
to 2010, the number of farm vehicles registered with 



the Ministry of Transportation increased by 90%, 
while in the decade ending in 2006, the number 
of farmers and the number of farms decreased by 
15%. The Ministry informed us that it had discussed 
the issue of the rise in number of registered farm 
vehicles with the Ministry of Transportation at the 
fraud prevention working group in November 2009, 
but no corrective action had been taken by the time 
we completed our audit fieldwork. 



RECOMMENDATION 2 



To help assess the appropriateness of vehicles 
exempted from testing and the geographical 
area covered by the Drive Clean program, the 
Ministry of the Environment should: 

• review initial pass/fail rates and evaluate 
estimated vehicle emissions by model year; 

• formally analyze the impact of excluding 
all light-duty vehicles except those in the 10 
larger municipalities in the Windsor-Quebec 
City corridor; and 

• work with the Ministry of Transportation 
on a strategy for verifying the legitimacy of 
farmers' vehicle registrations. 

MINISTRY RESPONSE 

The Ministry agrees that reviewing pass/fail 
rates and emissions by model year is important. 
The Ministry will continue to collect emis- 
sions test data to ensure a sufficient data set is 
assembled to conduct accurate analysis and will 
also continue to monitor the appropriateness of 
the geographical area covered by the program. 
In addition, the Ministry will work with the Min- 
istry of Transportation to evaluate its policies for 
registering farm vehicles. 



CONDITIONAL PASS 

In principle, the Drive Clean program requires 
vehicles that fail the emissions test to undergo 
repairs so that emissions are once again below the 
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ministry-prescribed limit. But to alleviate some 
of the financial burden on vehicle owners, the 
Ministry has implemented a $450 repair cost limit. 
That is, vehicle owners are allowed to defer emis- 
sion system repairs entirely or partly. If a single 
repair is required that would cost more than $450, 
that repair need not be done. If multiple repairs 
are required that together total more than $450, 
only repairs that fall within the $450 limit need to 
be done. Many other North American jurisdictions 
have a similar feature in their vehicle emissions 
inspection and maintenance programs. Owners 
who choose to defer repairs costing $450 or more 
are given a conditional pass, which allows them to 
renew their vehicle registration but not to transfer 
ownership. The vehicle can then be driven for two 
more years without the required repairs — in other 
words, until the next required emissions test, when 
they can again get a conditional pass assuming the 
repair estimate has not changed. 

More emissions reduction benefits could be 
achieved by eliminating or raising the repair 
cost limit, so that all required repairs would be 
performed. To that end, the Ministry raised the 
repair cost limit from $200 in 1999 to $450 in 2001 
for part of the geographical area covered by the 
program, and in 2004, the $450 limit was extended 
to the entire area. In order to receive a conditional 
pass, the vehicle owner must have a ministry- 
accredited Drive Clean facility determine what 
repairs need to be made, and have obtained a repair 
estimate exceeding $450. 

In 2011, conditional passes were issued to 18,000 
vehicles. We noted the following with respect to 
conditional passes under the present system: 

• We found that 7% of the vehicles that received 
a conditional pass in 2011 had no repairs 
done to their emissions control systems before 
being issued the conditional pass. For vehicle 
owners in that year who got a conditional pass 
but did have partial repairs done, the average 
repair cost was only $255. 

• Under the existing system, there is a risk that 
technicians could inflate actual or estimated 



repair costs in order to fraudulently obtain 
conditional passes for their customers without 
being detected. This problem will be mitigated 
starting in 2013, when the program adopts 
a new testing method. Under the existing 
testing method, repair technicians determine 
what repairs are needed by a failed vehicle, 
whereas under the new test method, the 
vehicle's on-board computer system will 
specify the problem. As well, the Ministry 
plans to have the service provider monitor 
the estimated and actual repair costs entered 
by the technician against a standard industry 
price list; this approach will enable it to flag 
unreasonable repair costs on a real-time basis. 

• In order to assess whether partial repairs are 
of any benefit to the province's air quality, we 
compared the emission readings before and 
after partial repairs were done on vehicles 
that were issued with conditional passes in 
2011. We noted that for 50% of these vehicles, 
emission readings for at least one of the pol- 
lutants measured got worse after the repairs, 
to the point where they exceeded emission 
limits. For 25% of vehicles, emission readings 
got worse for all pollutants measured. 

• The appropriateness of the repair cost limit 
has been in question since 2005, when an 
external consultant recommended increasing 
it to $600. The consultant also noted that the 
current $450 cap represents the same amount 
recommended in 1992 by the U.S. Environ- 
mental Protection Agency to vehicle inspec- 
tion and maintenance programs operating at 
that time in the United States. In our review 
of practices in other jurisdictions, we found 
that almost half of North American jurisdic- 
tions either have no repair cost limits or have 
a higher repair cost limit than Ontario's. For 
example, British Columbia sets different 
limits based on the vehicle's age, starting 
from $500 to unlimited. We also noted that 
from 2006 through 2010, the percentage of 
vehicles that pass a retest after any repairs 
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had been made dropped steadily each year, 
while the percentage of vehicles that received 
a conditional pass increased steadily. In 2010, 
56% of vehicles passed a retest (down 10% 
from 2006), and 39% received a conditional 
pass (up 13% from 2006). This pattern could 
indicate either that any repairs that were done 
are increasingly not effective or that the $450 
repair cost limit is too low. In addition, in the 
Emissions Benefit Analysis report for 2010, 
the consultant noted that the most commonly 
recommended repair involved servicing the 
catalytic converter, which was repaired or 
replaced in only one-third of cases. The con- 
sultant speculated that this could be because 
the maximum $450 repair cost limit was too 
low to cover this type of repair. 
Ontario requires vehicles that receive a 
conditional pass to be retested only every 
two years — in other words, on the same 
schedule as for vehicles that earn a regular 
pass. In comparison, 13 other North American 
jurisdictions require any vehicle that receives 
a conditional pass to be retested annually 
until the problem has been fixed. In 2005, the 
Ministry's external consultants recommended 
annual testing for vehicles that failed their 
initial emissions test, but this recommenda- 
tion was never implemented. 
Ontario allows an unlimited number of con- 
ditional passes to be issued for any particular 
vehicle. In comparison, five other North Amer- 
ican jurisdictions allow only one conditional 
pass to be issued during a vehicle's lifetime. 
The Ministry informed us that unlimited condi- 
tional passes are necessary to lessen the finan- 
cial burden for those who cannot afford to fully 
repair or replace a grossly polluting vehicle. 



RECOMMENDATION 3 



To help ensure that polluting vehicles are 
repaired once emission problems are identified, 
the Ministry of the Environment should consider: 



• increasing or eliminating the repair cost limit; 

• requiring vehicles that receive a conditional 
pass to be retested annually rather than 
biennially; and 

• limiting the number of conditional passes 
allowed over a vehicle's lifetime. 

MINISTRY RESP0NSI 

Less than 1% of all vehicles tested in the program 
use the repair cost limit. Starting in January 
2013, the new program will better identify emis- 
sions problems and prioritize the repairs needed, 
making more effective use of the $450 limit. 

The Ministry scans other jurisdictions and 
reviews its practices regularly. Once the new 
program is in place, we will continue to con- 
duct jurisdictional reviews, which will include 
reviewing repair costs and conditional passes, 
to support our analysis and evaluation of the 
program. 



EMISSIONS TEST METHODS 

The methods used to test emissions depend primar- 
ily on the type of vehicle and how it is powered: 

• Light-duty gasoline-powered vehicles, which 
are by far the majority of the vehicles in the 
province, are tested on a dynamometer — a 
kind of treadmill that simulates actual driving 
conditions. The Drive Clean facility's techni- 
cian inserts a probe into the vehicle's tailpipe; 
the probe, which is connected to a computer, 
measures the concentrations of various gases 
in the tailpipe's emissions. 

• Heavy-duty non-diesel vehicles, and certain 
light-duty vehicles that cannot be tested on 

a dynamometer for safety reasons, are tested 
by the two-speed idle method. This testing 
method measures fewer gases, and also has 
less stringent emissions limits than the dyna- 
mometer test. 
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• Heavy-duty diesel vehicles are tested using 
the opacity test method, where smoke density 
is measured by a smoke sensor. 

• Light-duty diesel vehicles are inspected vis- 
ually for emissions. 

Starting on January 1, 2013, Ontario will adopt 
a new on-board diagnostic (OBD) testing method 
for light-duty vehicles manufactured after 1997. All 
light-duty vehicles manufactured after 1997 have a 
built-in OBD system that continuously checks the 
condition and operation of key emissions control 
components and emissions-related systems in the 
vehicle. If the built-in system finds an emissions- 
related problem, it illuminates the "check engine" 
light on the vehicle's dashboard and stores one or 
more "diagnostic trouble codes" in the vehicle's 
computer. Thus, any vehicle with an illuminated 
"check engine" light may have an emissions prob- 
lem, and the specific diagnostic trouble code(s) 
help to identify this as well as the cause or causes 
of the problem. Once the vehicle is connected to 
the new testing equipment it will retrieve the prob- 
lems and produce a "fail" result. All other North 
American jurisdictions with vehicle inspection and 
maintenance programs have been using this testing 
method for a number of years. 

The OBD testing method will require the Drive 
Clean facilities to use new testing equipment. The 
test certificate issued to vehicle owners will also 
have a different format. This certificate, unlike the 
ones currently being given out, will not include 
emission limits and actual emissions levels. Instead, 
it will show whether the "check engine" light is 
illuminated and which vehicle diagnostic trouble 
code or codes were detected. 

The emission limits set in the vehicle's OBD 
component are more stringent than those in the 
dynamometer. The Ministry informed us that OBD 
testing could cause a vehicle to fail when emis- 
sions exceed 1.5 times the federal limit, whereas a 
dynamometer will not cause a vehicle to fail until 
emissions are much higher — an estimated five to 
eight times the federal limit. As a result, the Min- 
istry expects initial failure rates to double from 5% 



to 10% in 2013, which in its view brings Ontario 
in line with other jurisdictions. In other words, the 
Drive Clean program could fail up to 270,000 light- 
duty vehicles in 2013, compared with the 135,000 
that failed in 2010. When British Columbia adopted 
OBD testing, its initial failure rate went from about 
12% to about 15%. 

However, older vehicles, which typically are 
the worst polluters, cannot be tested by the new 
testing method, because light-duty vehicles manu- 
factured in Canada before 1998 were not equipped 
with OBD technology. In December 2011, almost 
600,000 vehicles (8%) registered in the geographic 
area covered by the program were built between 
1988 and 1997. The Ministry plans to have these 
vehicles, which are now being tested on a dyna- 
mometer (equipment that will be phased out when 
the OBD equipment is installed), tested by the two- 
speed idle method. Since the two-speed idle test 
method has less stringent emission limits than the 
old dynamometer test, the initial pass rate for these 
older vehicles will likely improve, even though 
there will be no real improvement in emission per- 
formance — and fewer of these older vehicles that 
require repairs will be identified. We researched 
the testing methods used in other North American 
jurisdictions for vehicles that are not equipped with 
an OBD system, and noted that more than half of 
these jurisdictions use a testing method that is more 
stringent than the two-speed idle method Ontario 
plans to use. The Ministry informed us that since all 
emissions testing equipment requires regular main- 
tenance, it will not require Drive Clean facilities 
to maintain both a dynamometer and OBD testing 
equipment. Consequently, older vehicles — which 
currently have both lower initial pass rates and less 
stringent limits — will be held to even less stringent 
limits, whereas newer vehicles — which currently 
have both high initial pass rates and more stringent 
emission limits — will be held to even stricter emis- 
sion limits. 
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Implementation of On-board Diagnostic 
Test Method 

External consultants first recommended OBD test- 
ing to the Ministry in the program's operational 
review in 2005 and again in 2007. One of these 
consultants noted that the costs of the dynamom- 
eter test method will outweigh its emission benefits 
by 2015. It further noted that the OBD test method 
would yield greater emission reductions from all 
vehicles combined than the dynamometer test 
method. OBD testing has been used in British Col- 
umbia since 2007 and was in place in all U.S. juris- 
dictions with vehicle inspection and maintenance 
programs by 2006. The improved testing method is 
being implemented eight years after the first oper- 
ational review that recommended this switch and 
six or seven years behind all other North American 
jurisdictions that have a vehicle inspection and 
maintenance program. The Ministry estimates that 
by 2015, 96% of the vehicles on Ontario's roads will 
be equipped with OBD technology. 

One benefit of being the last jurisdiction to 
adopt OBD testing is that the Ministry has been able 
to research the lessons learned in other jurisdic- 
tions that have already adopted the technology, 
particularly with regard to detecting possible fraud 
by testing facilities. To this end, we noted that the 
Ministry considered other jurisdictions' experi- 
ence in using devices (such as video cameras) and 
triggers (predetermined indicators of suspicious 
activity that set off warnings within the information 
system) to monitor program compliance by test and 
repair facilities. 

New Emissions Testing Equipment 

We requested that certain tests be performed on 
one of the new testing units that will be rolled out 
to all testing facilities in January 2013, and noted 
the following problems: 

• The new testing equipment has a camera that 
allows the Drive Clean facility to take photo- 
graphs of the vehicle identification number, 
licence plate and odometer so that this infor- 



mation can be stored in the system, allowing 
compliance staff at the private-sector service 
provider to verify later on that the vehicle 
being tested was in fact the one subject to the 
test. We noted that uploading photographs via 
a dial-up Internet connection (one of four pos- 
sible ways to connect the testing equipment to 
the program's information system — and prob- 
ably the only way available to testing facilities 
in certain remote areas) would abort the test 
and crash the testing equipment. 

• During an emissions test, the Drive Clean 
facility can manually change the record for 
the engine type of a vehicle from non-diesel to 
diesel, thus allowing the vehicle to undergo a 
simple visual test rather than the more strin- 
gent OBD test. 

• The new testing equipment accepts unreason- 
able odometer readings, which in turn will 
affect the accuracy of the emission benefit 
calculated. For example, we observed that 
the equipment accepted an odometer reading 
that was lower than that entered in a previ- 
ous test, and nevertheless generated a "pass" 
emissions report. 



RECOMMENDATION 4 



To optimize the benefits of the new on-board 
diagnostic testing technology, the Ministry of 
the Environment should ensure that appropriate 
technical testing is completed and problems are 
resolved before rolling it out to all Drive Clean 
testing facilities in January 2013. The Ministry 
should also monitor the potential impact of 
using the less reliable two-speed idle method 
for testing vehicles older than model-year 1998 
once the new on-board testing technology has 
been introduced. 

| MINISTRY RESPONSE 

The Ministry agrees that appropriate technical 
testing is important. In the last year, the Min- 
istry instituted a formal defect-management 
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process to ensure that technical testing and 
corrections on the new test equipment are 
completed. As well, the Ministry initiated field 
testing earlier this year in 30 facilities. Issues 
identified at the pilot sites were resolved. 

All of the issues identified in the Auditor's 
report regarding uploading photos, manual 
changes to the engine type and odometer read- 
ings will be resolved prior to the program roll- 
out in January 2013. 

We note the Auditor General's comments 
that vehicles built from 1988 to 1997 can only 
be tested using a tailpipe test. The Ministry has 
determined that the two-speed idle tailpipe test 
is a cost-effective solution for an ever-decreasing 
proportion of older vehicles (that is, an esti- 
mated 8% of all light-duty vehicles in 2013) . It is 
worth noting that maintaining the current dyna- 
mometer equipment would be cost-prohibitive 
for many Drive Clean facilities. 



MONITORING PROGRAM DELIVERY 

Compliance of Drive Clean Facilities 

Fraudulent test results have negative impacts 
on both the environment and vehicle owners: a 
fraudulent pass will permit a high-polluting vehicle 
to remain on the road, and a fraudulent fail will 
cost the vehicle owner for unnecessary repairs. 
As of December 2011, Ontario had almost 1,500 
testing and repair facilities for light-duty vehicles 
and more than 500 testing and repair facilities for 
heavy-duty vehicles. 

The Ministry outsources the monitoring of Drive 
Clean facilities for non-compliant or fraudulent 
activities to the private-sector service provider. 
The service provider uses a number of compliance 
techniques to monitor facilities' compliance with 
contract requirements and standard operating pro- 
cedures. Generally, the service provider analyzes 
data captured in the Drive Clean information sys- 
tem to detect occurrences of predetermined indica- 



tors of suspicious activity (called triggers), so that it 
can target those facilities that are most likely to be 
participating in non-compliant or fraudulent activ- 
ities. Facilities that set off these triggers are then 
audited using one of the following approaches: 

• telephone audits — unconcealed audits where 
the service provider's compliance staff call the 
facility to obtain certain documents for review; 

• overt audits — annual unconcealed audits 
where compliance staff check various admin- 
istrative matters, such as whether the facility 
is using appropriate testing equipment; and 

• covert audits — ad hoc, undercover "secret 
shopper" audits where compliance staff pose 
as customers to try to uncover fraudulent 
test practices. 

Audits are designed to uncover cases of non- 
compliance, such as charging test fees that are 
higher than allowable, falsifying test results, 
and testing vehicles using a more lenient testing 
method than the required method. We noted that 
covert audits can identify some of the most serious 
program offences and generally detect more cases 
of non-compliance than any other type of audit. In 
each of the last three years, covert audits have dis- 
covered non-compliance in about 60% of the audits 
conducted, compared to about 40% with other 
types of audits. 

However, the use of covert audits has dramatic- 
ally diminished. The latest contract required the 
service provider to perform 1,400 covert audits 
annually at Drive Clean facilities that test light-duty 
vehicles until June 2011, none between July and 
December 2011, and, going forward from 2012, 
a very small fraction of the initial requirement of 
1,400. No covert audits are required for facilities 
that test heavy-duty vehicles. In its 2009 Treasury 
Board submission requesting approval to consoli- 
date all outsourced program services, the Ministry 
indicated that the number of covert audits was 
expected to be at most 50% of what was done prior 
to July 2011. 

To compensate for the reduction of covert 
audits, the Ministry plans to implement a real-time 
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monitoring function in its Drive Clean information 
system starting in 2013, to coincide with the new 
OBD testing method. This means that when trig- 
gers for certain suspicious activities are set off, the 
information system will immediately lock out the 
testing equipment, and compliance staff will have a 
face-to-face conversation with Drive Clean facility 
staff using a video camera attached to the testing 
equipment. According to the Ministry, Ontario may 
be the first North American jurisdiction to employ 
this kind of cost-effective compliance technique. 

Triggers are primarily designed to ensure that 
the vehicle tested is the one subject to the test and 
to allow compliance staff to monitor patterns that 
suggest inappropriate testing and repair practices. 
We noted that although the number of triggers will 
increase in 2013 from about 30 to 50, only 1 in 4 
of these triggers will be monitored on a real-time 
basis; the rest will be recorded in the information 
system, and compliance staff will follow up on them 
at a later date. Because these triggers were not 
operational at the time of our audit, we could not 
assess whether they will be as effective as covert 
audits done in the past. 

Consequences of Non-compliance 

The private-sector service provider brings cases 
of non-compliance to the Ministry's attention on 
a weekly basis, so that the Ministry can determine 
the appropriate penalty depending on the type and 
frequency of the offence. For calendar years 2009 
and 2010, the service provider's audit activities 
identified about 3,000 cases of non-compliance 
annually, and the Ministry took remedial action in 
about 15% of these cases. The Ministry explained 
that it imposes penalties only on non-compliance 
cases found through covert audits, and that it 
imposed penalties in half of those cases. For non- 
compliance identified through other types of audits, 
the Ministry stated that it expects that the appropri- 
ate remedial action will be taken by the service pro- 
vider, since those cases are usually minor in nature. 
However, the Ministry did not track remedial action 



imposed by the service provider. The various types 
of remedial actions, in order of severity, as well as 
the frequency with which each type of action was 
taken by the Ministry in 2009 and 2010, were as 
follows: 

• phone call (11%); 

• note to file (27%); 

• warning letter (47%) ; 

• suspension (11%); and 

• termination (4%) . 

Because the service provider conducted far 
fewer covert audits in 2011 than in previous years, 
the number of remedial actions applied in 2011 
was about 60% lower than the average number of 
actions taken in 2009 and 2010. 

We noted that the Ministry does prosecute ser- 
ious offences. Between 2009 and 2011, the Ministry 
convicted 29 individuals and seven Drive Clean 
facilities for offences relating to Drive Clean fraud, 
and collected fines totalling $446,000. 

Ministry staff informed us that they typically 
apply more lenient penalties than those recom- 
mended in the remedy schedule in the Ministry's 
procedural manual. For example, when a Drive 
Clean facility is caught overcharging vehicle owners 
for Drive Clean tests, Ministry policy requires that 
the facility be suspended for a first offence and 
terminated for a repeat offence, but we noted that 
between January 2005 and February 2012, only 
one in every 34 facilities found to be overcharging 
were in fact suspended or terminated. We analyzed 
the nature of the overcharges, and while many 
cases of non-compliance related to misapplying 
sales tax on the test fees, a few facilities — some of 
which were repeat offenders — had more question- 
able overcharges or engaged in inappropriate fee 
collection practices but were not suspended or ter- 
minated. We could not assess the appropriateness 
of penalties for more serious non-compliance items, 
such as falsifying test results or testing vehicles 
using an inappropriately lenient testing method, 
because the Ministry does not possess complete 
data for all non-compliance items identified and the 
resulting penalties applied. We also found almost 
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40 cases of non-compliance (where, for example, 
facilities were caught selling vehicle owners 
unnecessary services or repairs, or testing vehicles 
using an inappropriate test method) for which no 
penalty had yet been applied even after one to two 
years had passed. 

Our analysis of audit results showed that the 
compliance rate remained the same between 2009 
and 2011, which could indicate that the penalties 
applied in previous years may not be having the 
desired deterrent effect. 

Duplicate Test Certificates 

In our 2004 audit of the Ministry's Air Quality Pro- 
gram, we identified 3,200 uniquely numbered test 
certificates that had been presented at Ministry of 
Transportation (MTO) offices (now ServiceOntario) 
more than five times each. In 2006, the Ministry 
introduced a security code on all certificates 
designed to prevent fraudulent certificate use. 
Unless both the test certificate number and a valid 
security code match the vehicle identification num- 
ber, ServiceOntario staff will not accept the pass test 
certificate for renewing the vehicle's registration. 
While technical issues do arise from time to time, 
they are infrequent, and we concluded that the 
Ministry has taken appropriate steps to address the 
concerns we raised in 2004. 



RECOMMENDATION 5 



To maintain the integrity of the Drive Clean pro- 
gram, the Ministry of the Environment should: 

• use compliance rates to periodically evalu- 
ate the appropriateness of the mix of audit 
compliance tools, especially given the 
planned substantial decrease in covert audit 
activities; and 

• maintain complete data for all non-compliance 
items identified and their related penalties, 
and ensure that the penalties applied are 
appropriate, consistent and timely. 




The Ministry agrees with the recommendation 
of the Auditor General. 

The modernized Drive Clean program 
includes the transformation of the current 
reactive audit approach to a proactive real-time, 
risk-based audit process. Through the use of 
state-of-the-art, web-based technology, Drive 
Clean auditors will be able to identify potential 
non-compliance issues and speak directly with 
Drive Clean technicians, review test informa- 
tion, and provide guidance as tests and repairs 
are being conducted. Should a potential non- 
compliant event be detected, the auditor will 
interrupt the test and promptly address the 
issue. The Ministry will continue to use a variety 
of enforcement activities, including education 
and outreach, suspensions, and revocations of 
operating certifications. 

Our new audit process, which includes a 
suite of audit tools, will begin in January 2013. 
The Ministry will continue to monitor the pro- 
cess and collect compliance data to determine 
the appropriateness, consistency and effective- 
ness of both its compliance tools and penalties. 



CONTRACT MANAGEMENT 

Procurement of Service Provider 

Before June 2011, the Ministry had six contracts 
with three different service providers to deliver six 
distinct program services: ensuring quality control 
at Drive Clean facilities for the light-duty and the 
heavy-duty vehicle program components, training 
vehicle inspectors and repair technicians, operating 
a call centre to provide technical support to Drive 
Clean facilities and handle public comments, print- 
ing and distributing program materials to Drive 
Clean facilities, and developing and operating an 
information system to support all program func- 
tions, including linking to the Ministry of Transpor- 
tation's licensing system. 
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In 2011, after a competitive procurement pro- 
cess, the Ministry began outsourcing all program 
services to a single service provider — the one that 
had been providing four of the six services since the 
program's inception. Under the current contract, 
this service provider will also supply and service 
the new OBD emissions testing equipment. The 
contract, which expires in June 2018, includes an 
option to renew annually for up to three additional 
years. Consolidating all program services under one 
contract with a single service provider was recom- 
mended by an external consultant in 2007. The 
Ministry's decision to do so was supported by Treas- 
ury Board in November 2009, and this practice is 
consistent with that in several other jurisdictions 
with similar programs. 

The Ministry hired a fairness commissioner to 
oversee and evaluate the procurement process for 
the new contract. He concluded that "appropriate 
procurement practice was used throughout" the 
process. Perhaps most importantly, the new con- 
tract's expected annualized price will be 40% lower 
than the sum of what the previous six contracts cost 
each year. According to the contract, the service pro- 
vider can receive up to $62 million from the prov- 
ince over the contract term, or cancel on six months' 
notice, and will also receive amounts directly from 
facility operators for training their inspectors and 
repair technicians, facility accreditation, and supply 
and maintenance of the new OBD testing equip- 
ment. In this regard, the Ministry awarded the 
service provider the exclusive right to subcontract 
the design, building and testing of the new testing 
equipment, and to be that equipment's sole supplier 
to Drive Clean facilities. Based on information avail- 
able at the time of our audit, the service provider 
plans to sell the testing equipment to facilities for 
up to $20,000 plus up to $350 per month for main- 
tenance. The Ministry indicated that, to assess the 
reasonableness of the purchase price and mainten- 
ance costs that facility operators will have to pay, 
it had compared these prices to those offered by a 
supplier in another jurisdiction, and found that the 
costs were generally comparable over the expected 



life of the program. However, we found that the 
Ministry compared Ontario pricing with that in only 
one other jurisdiction, which coincidentally uses the 
same service provider as Ontario. 

Monitoring Service Provider Activity 

The private-sector service provider must meet 
certain service levels in order to receive full con- 
tract payment. The agreement indicates that if the 
service provider fails to meet deliverables by the 
required dates, the Ministry can withhold specified 
amounts from subsequent payments to the service 
provider as a penalty. In negotiating the contract 
with the new service provider, we felt the Ministry 
demonstrated good foresight by including more 
than 70 deliverables in various schedules, many 
of which were aimed at ensuring a successful 
transition to the new testing method and related 
processes by January 1, 2013. 

We assessed about 50 of the most significant 
deliverables to ensure that the Ministry was mon- 
itoring them and found that almost 80% of the 
deliverables deadlines were met. For the remaining 
20%, the deadlines were extended or there was a 
lack of documented evidence that the service was 
ultimately provided. For the sample of deliverables 
we tested, penalties were generally applied when 
deliverables were late, unless their delivery dates 
were extended, in which case no penalty was 
applied. At the time of our audit, we could not 
assess how ready the Ministry will be to roll out the 
new testing method and related processes by Janu- 
ary 1, 2013. 

We also noted that the Ministry sometimes 
relies on the service provider's claim that it has 
met certain requirements but does not verify the 
reported information. For example, service provider 
staff is required to be up to date on the program, 
but the Ministry does not review the results of 
tests designed to verify that the service provider's 
employees have maintained current program know- 
ledge. In another example, the Ministry does not 
review summary reports of call centre data to verify 
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the service provider's claim that calls were answered 
within the prescribed time. In both of these cases, 
the Ministry has made payments to the service pro- 
vider in full without checking whether the service 
provider had indeed met the service requirements. 

Complaints 

Complaints are received and resolved by the call 
centre that is operated by the private-sector service 
provider. The call centre receives on average 100 
complaint calls per month from the public. In a 
typical month, more than half of all callers report 
a smoking vehicle, 30% express dissatisfaction 
with a Drive Clean facility (for example, because of 
repair misconduct or because their vehicle failed its 
emissions test at one testing facility but passed at 
another), and 12% voice general complaints about 
the program (regarding, for example, the cost of 
the emissions test or the required frequency of test- 
ing) . We were informed that Drive Clean facilities 
that are the subject of complaints are considered for 
audit by the service provider. 

Under the terms of the agreement, the Ministry 
can withhold payments to the service provider if 
call handling quality is substandard — that is, if the 
call centre does not provide accurate, complete and 
current program information to callers 95% of the 
time. To assess service quality, the Ministry must 
review a specified minimum number of calls each 
month. However, we found no documented evi- 
dence that the Ministry had monitored the required 
number of calls. Without such oversight, the Min- 
istry can't reliably assess performance and withhold 
payment when warranted. 

We also noted that between July 2011 and 
February 2012, almost 160 calls were logged as 
unresolved. The Ministry advised us that these call- 
ers' complaints were indeed resolved, but because 
call centre staff were unfamiliar with the call sys- 
tem, the handling of the calls had been erroneously 
recorded as unresolved. 



RECOMMENDATION 6 



To help ensure that the private-sector service 
provider meets contractual obligations in deliv- 
ering the Drive Clean program, the Ministry of 
the Environment should adequately monitor the 
delivery of all services, including periodically 
verifying reported service levels achieved. 



The Ministry agrees with the recommendation 
of the Auditor General. 

In order to monitor the service provider's 
performance, the Ministry, along with the 
service provider, is developing a Service Level 
Tracking application to continuously monitor all 
service levels and apply penalties as appropri- 
ate. If service-level standards are not met, the 
service provider must pay a monetary penalty. 

The Ministry has also instituted a business 
process to ensure that all calls received by the 
Drive Clean Office are reviewed, followed up 
and closed. 



PROGRAM REVENUE 

Fee Collection 

Owners of light-duty vehicles pay $35 for an initial 
test and $17.50 for a retest after any repairs have 
been made, while owners of heavy-duty vehicles 
pay market rates for the test. The Ministry is entitled 
to receive $11.67 for light-duty vehicles and $15 
for heavy-duty vehicles, but only if the vehicle 
receives a pass or conditional pass. (The govern- 
ment receives no revenue from failed tests or retests, 
because it does not want to collect multiple test fees 
from vehicle owners. Vehicle owners still have to pay 
the test fee imposed by the testing facility.) 

At the time of our audit, Drive Clean facilities 
were forwarding the Ministry's portion of test rev- 
enue to the service provider, who in turn submitted 
all revenues received to the province. Starting in 
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January 2013, the facilities will directly remit test 
revenues to the government without the involve- 
ment of the service provider. 

To ensure that the Ministry receives all test 
revenues owing to it, each month the Ministry rec- 
onciles payments received from the service provider 
with the number of pass or conditional pass results 
indicated by the Drive Clean information system. 

However, due to instances of data corruption 
and system communications errors, the information 
system may not contain all test results. Therefore, 
to identify omissions in the system, the Ministry 
runs a monthly exception report that reconciles 
Drive Clean test certificate numbers processed by 
ServiceOntario using the Ministry of Transportation 
licensing system with those recorded in the Drive 
Clean information system, and any extra tests are 
validated and added to the system. This is an effect- 
ive detective control process, and we noted that 
in 2011, this process detected about 340 tests that 
were registered at ServiceOntario but not initially 
recorded in the Drive Clean information system. 

Revenue Neutrality 

Test revenues collected by the Drive Clean program 
are considered a user fee, not a tax. However, 
according to a 1998 Supreme Court of Canada deci- 
sion, user fees must have a reasonable relationship 
to the cost of the services provided. In other words, 
a user fee cannot exceed the cost to the govern- 
ment of providing the service. Otherwise, a court 
could determine that the amount of the excess fee 
is really an unlawful tax and therefore repayable. 
The Ministry of Finance's 2006 Costing and Pricing 
Policy indicates that where fees or other charges are 
collected for services offered to the public, pricing 



should be based on consideration of the full costs of 
delivering the service. 

One of the Drive Clean program's goals is to 
achieve revenue neutrality over the program's 
life. By the end of the 2010/11 fiscal year the 
present value of cumulative revenues collected 
and expenses incurred since 1999 had reached 
the break-even point. However, revenues are now 
exceeding program expenses. In December 2011, 
the Ministry determined that (based on a net 
present value calculation) the program is expected 
to realize an accumulated surplus of $11 million 
by the end of the 2011/12 fiscal year and $50 mil- 
lion by the time the current consolidated contract 
with the service provider ends in 2018. At the time 
of our audit, the Ministry informed us that it had 
received advice and was considering options for 
becoming compliant with both the 2006 Costing 
and Pricing Policy and the 1998 Supreme Court of 
Canada decision. 

OTHER MATTER 

Vehicle Retirement Program 

The Ministry provides in-kind contributions 
(through promoting the program to increase public 
awareness of its existence) to a national vehicle 
scrappage program under which a vehicle owner 
who donates an older vehicle that meets certain 
conditions is given a charitable donation receipt 
of at least $100. Other jurisdictions have similar 
programs, but with more generous incentives. For 
instance, California's car scrappage program offers 
a minimum of $1,000, and up to $1,500 for low- 
income earners, for unwanted vehicles that meet 
certain eligibility guidelines. 



Chapter 3 
Section 




Ministry of Education 



Education of 
Aboriginal Students 



Background 




According to the most recent Statistics Canada 
census data (2006), there are 242,000 Aboriginal 
people living in Ontario, primarily comprising 
three distinct groups: First Nation (158,000), 
Metis (74,000) and Inuit (2,000). The remaining 
8,000 Aboriginal people classified themselves as 
"other." Aboriginal peoples in Ontario have diverse 
languages, cultures and traditions. The census also 
identified that approximately 47,000 First Nations 
people live on reserves in Ontario; these are lands 
set aside for the use and benefit of a specific band 
or First Nation. There are a total of 133 First Nation 
communities in Ontario, each of which has its own 
government or tribal council. The delivery of edu- 
cation through schools on reserve is the responsibil- 
ity of the First Nation and the federal government. 
The federal government is financially responsible 
for the education of First Nation students living on 
reserve, whether these students attend First Nation 
or provincially operated schools. 

As shown in Figure 1, the 2006 census (the most 
recent definitive data on Aboriginal education) 
identified that only 62% of Aboriginal adults had 
graduated from high school as compared to 78% of 
the general population, a gap of 16%. The academic 
achievement gap is even more pronounced — as 



high as 50% — for the youngest adult age group 
(aged 20-24) . Only 39% of the First Nation people 
living on reserve in this age group had graduated 
from high school. This issue is all the more import- 
ant considering that 46% of First Nation people 
are under the age of 25, as compared to the overall 
population where 32% are less than 25 years old. 

The Ministry of Education (Ministry) adminis- 
ters provincially funded elementary and second- 
ary education in Ontario and is responsible for 
developing the curriculum, setting requirements for 
student diplomas and providing funding to school 
boards. In 2006, the Ministry identified Aborig- 
inal education as one of its key priorities, with a 
focus on closing the gap in academic achievement 
between Aboriginal and non-Aboriginal students by 
the year 2016. To meet this challenge, the Ministry 
released various policy documents and funded a 
number of initiatives intended to improve outcomes 
for Aboriginal students. Since 2006, the Ministry 
has provided $170 million in funding to improve 
academic achievement levels for Aboriginal stu- 
dents and increase awareness of Aboriginal cultures 
for all students. 

In January 2006, the Aboriginal Education 
Office (AEO) was created to provide co-ordination 
and leadership on Aboriginal education issues and 
initiatives. The AEO works in collaboration with 
Aboriginal communities and organizations, school 
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Figure 1: The Gap in Aboriginal High School Graduation Rates for Adults and Young Adults 

Source of data: 2006 Statistics Canada census and Ministry of Education 
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All Aboriginal 


62 


16 


66 


23 
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Note: The percentage graduated is the percentage of the population group with a high school diploma. The gap in graduation rates equals the 
percentage of all people graduated in Ontario minus the percentage graduated in the Aboriginal group. 



boards, other ministries and the federal govern- 
ment. Currently, the AEO employs 12 staff, includ- 
ing five regional education officers who work with 
Ontario's 72 school boards. 

Research indicates that many Aboriginal stu- 
dents are faced with significant challenges that 
impact their achievement levels in school, such 
as high rates of poverty, substandard housing and 
poor nutrition. Equally important, many of these 
students live in areas with little prospect of future 
employment, which is often a demoralizing factor 
in taking their education seriously. In 2008, Health 
Canada reported that there is an extraordinarily 
high and increasing rate of suicide among Aborig- 
inal youth. Although the Ministry of Education 
cannot resolve all of these issues, it can promote 
achievement in school to help Aboriginal students 
overcome these challenges. 



Audit Objective and Scope 



The objectives of our audit were to assess whether 
the Ministry and selected school boards have 
adequate procedures in place to: 

• identify and implement initiatives to improve 
Aboriginal student achievement and to meas- 
ure and report on the effectiveness of those 
initiatives; and 



• ensure that transfer payments intended for 
Aboriginal education initiatives are spent for 
the purposes intended and allocated on the 
basis of student need. 

Senior management reviewed and agreed to our 
audit objectives and associated audit criteria. 

Our audit work was conducted at the Ministry's 
head office, primarily in the Aboriginal Education 
Office. We also conducted audit work at selected 
school boards and at a sample of elementary and 
secondary schools in these boards. The three 
boards we visited were Algoma District School 
Board (Sault Ste. Marie), Kawartha Pine Ridge 
District School Board (Peterborough) and Lakehead 
District School Board (Thunder Bay) . 

Our audit work included reviewing and analyz- 
ing ministry files, administrative directives, policies 
and procedures, and interviewing ministry staff. 
We also met with school board staff, including 
supervisory officers, principals and teachers. We 
researched Aboriginal education practices in other 
jurisdictions and solicited the opinions of First 
Nation education managers, First Nation Elders, the 
Chiefs of Ontario and the Metis Nation of Ontario. 
Our audit also included a review of activities of 
the Ministry's Internal Audit Services Branch 
relating to Aboriginal education. We reviewed the 
branch's recent reports and considered its work and 
any relevant issues identified when planning our 
audit work. 
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Summary 




In 2006, the Ministry identified Aboriginal educa- 
tion as one of its key priorities with a focus on meet- 
ing two primary challenges by 2016: to significantly 
improve achievement among First Nation, Metis 
and Inuit students, and to close the significant 
student achievement gap between Aboriginal and 
non-Aboriginal students. The Ministry also wanted 
all students to have an appreciation of Aboriginal 
cultures. In order to help achieve these objectives, 
the Ministry designed an overarching policy 
framework that endeavoured to identify Aboriginal 
students in Ontario, guide the development of 
appropriate support programs, periodically assess 
academic achievement and report on the progress 
made in implementing the framework. However, 
the Ministry needs to more actively oversee the 
implementation of this framework to demonstrate 
what, if any, progress has been made since 2006 in 
improving achievement among Aboriginal students 
and closing the achievement gap between them and 
other students in Ontario. 

Some of our more significant observations 
regarding Aboriginal education in Ontario were: 
• In 2007, the Ministry released the Ontario 
First Nation, Metis and Inuit Education Policy 
Framework (Framework), which includes 10 
performance measures intended to gauge the 
success of ministry, school board and school 
efforts to improve the academic achievement 
of Aboriginal students. Although the Ministry 
has reported some success on several of these 
measures, five years after the release of the 
Framework the Ministry has not quantified 
any of the 10 performance indicators so that 
progress could be objectively measured, nor 
has it required that the school boards individ- 
ually evaluate and report on these perform- 
ance measures. None of the three boards we 
visited had done so. 



Since the Ministry did not have an accurate 
and reliable mechanism to identify Aboriginal 
students in the Ontario school system, in 2007 
it released Building Bridges to Success for First 
Nation, Metis and Inuit Students. This policy 
guide assists school boards in developing 
Aboriginal student self-identification policies 
whereby students are given the option to 
formally identify themselves as First Nation, 
Metis or Inuit. At the time of our audit, less 
than half of the estimated Aboriginal student 
population in Ontario had self-identified. 
Improved progress on this self-identification 
strategy would enable the Ministry and 
boards to better target funding and periodic- 
ally benchmark the academic progress of 
Aboriginal students. 

Five years after announcing its intention to 
significantly close the gap in achievement 
between Aboriginal and non-Aboriginal stu- 
dents, the Ministry has still not established a 
baseline or even a starting point from which 
to measure the gap. Therefore, we requested 
that the Ministry provide us with the most 
recent student data, including Grade 10 credit 
accumulation for self-identified Aboriginal 
students. Accumulation of credits toward 
graduation is one of the primary indicators 
of student success in high school. We noted 
that only 45% of the self-identified Aboriginal 
Grade 10 students were on track to graduate 
from high school, compared to 74% for all stu- 
dents in the Grade 10 population. This brings 
into question the Ministry's ability to meet its 
goal of significantly closing the achievement 
gap between Aboriginal and non-Aboriginal 
students by 2016. 

Even though an Aboriginal organization 
engaged by the Ministry in 2009 reported 
that consistent implementation of the 
Framework is necessary to make systemic 
change, we found that it was largely left 
up to each board to decide how it would 
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implement the Framework. The emphasis 
placed on this varied at the three boards we 
visited. For example, one of the school boards 
we visited had successfully implemented the 
self-identification policy as virtually all of its 
Aboriginal students had self-identified. This 
board was using student achievement data to 
identify challenges unique to its Aboriginal 
population and was implementing specific 
strategies to address these challenges. 
• Although the early education of many First 
Nation students is provided by reserve 
schools, which are the financial responsibility 
of the federal government, many of these stu- 
dents eventually transfer into the provincial 
system. Given limited per-student funding, 
on-reserve schools have generally not been 
able to provide the quality of education found 
in provincial schools; studies suggest that 
these students may be several grade levels 
behind when they transfer into the public sys- 
tem. Although the Ministry does not measure 
this, our analysis of Education Quality and 
Accountability Office (EQAO) data found that 
only half of on-reserve students attending 
provincially funded schools passed the Grade 
10 Ontario Secondary School Literacy Test 
in the 2010/11 school year, compared to the 
provincial average of 83%. 



1 OVERALL MINISTRY RESPON 




Too many Aboriginal students do not achieve 
the academic success we want for all students. 
Addressing this education attainment gap is a 
top priority for the government of Ontario. We 
are pleased that the Auditor General shares 
our commitment to improving educational out- 
comes for Aboriginal students. 

The Ministry of Education (Ministry) and 
boards, working with First Nation, Metis and 
Inuit communities, share the view that signifi- 
cant progress has been achieved, and that con- 
ditions for future success have been established 



through progressive collaboration and specific 
supports. Some boards have made progress in 
implementing supports for Aboriginal students 
and recent evidence reveals specific instances 
where the achievement gap has been reduced. 
The Ministry agrees with the spirit of the 
Auditor General's recommendations and will 
continue to develop and implement strategies 
targeted to improve Aboriginal student success 
within the context, philosophy and approach 
outlined in the Ontario First Nation, Metis and 
Inuit Education Policy Framework. 

The Ministry's Framework is based on 
Ontario's New Approach to Aboriginal Affairs, 
2005, which charts a new course for a construct- 
ive, co-operative relationship with the Aborig- 
inal peoples of Ontario to develop productive 
partnerships, collaborate on key initiatives and 
achieve real progress on shared goals. 

The Ministry, with its partners, has put in 
place the necessary foundation for success in 
keeping with our overall student achievement 
strategy: building relationships and capacity 
within the sector; designing and developing 
data collection mechanisms; and analyzing and 
implementing evidence-driven strategies to sup- 
port advancements in student achievement. 

When considering evidence-based strategies 
and reporting, it is important to understand 
Aboriginal peoples' perspectives on the school 
system, including the intergenerational mistrust 
resulting from residential school experiences. 
Ontario respects Aboriginal and treaty rights 
protected by Section 35 of the Constitution Act, 
1982. This is why the collection of Aboriginal 
student data must continue to be voluntary 
and confidential. 

Moving forward, the Ministry will pursue a 
path that builds on demonstrated successes, in 
partnership with communities, school boards, 
the federal government and Aboriginal part- 
ners, to support First Nation, Metis and Inuit 
students across the province. 
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Detailed Audit Observations 



POLICY GOALS, IMPLEMENTATION 
PLANS AND PERFORMANCE MEASURES 

Policy Goals and Planning 

In June 2005, the government of Ontario released 
Ontario's New Approach to Aboriginal Affairs, which 
outlined a number of initiatives aimed at achiev- 
ing real progress on shared goals with Aboriginal 
people. In line with this approach, the Ministry 
identified Aboriginal education as one of its key 
priorities, with a focus on meeting two primary 
challenges by the year 2016: to improve achieve- 
ment among First Nation, Metis and Inuit students; 
and to close the gap between Aboriginal and 
non-Aboriginal students in literacy and numeracy, 
retention of students in school, graduation rates 
and advancement to post-secondary studies. 

In 2007 the Ministry released the Ontario First 
Nation, Metis and Inuit Education Policy Framework 



(Framework) . The Ministry considers the Frame- 
work to be the foundation for delivering quality 
education to all Aboriginal students who attend 
provincially funded elementary and secondary 
schools. Figure 2 shows the Ministry's three goals 
and the 10 performance measures it intends to use 
to gauge success, as outlined in the Framework. 

In order to be truly effective, the Framework 
recognized that it was necessary for the Ministry 
to develop an implementation plan to guide its 
activities and assist school boards in meeting the 
broad objectives in the Framework. Although the 
Ministry has engaged in some preliminary imple- 
mentation planning, five years after the release of 
the Framework, the Ministry has still not developed 
a formal implementation plan. In our opinion, such 
a plan should identify the key obstacles faced by 
Aboriginal students and outline specific activities 
to overcome various obstacles. This was exactly 
the process we found the Ministry had used in our 
recent audits of school safety programs and student 
success initiatives, both of which had achieved 



Figure 2: Ontario First Nation, Metis and Inuit Education Policy FrameworkGoa\s and Performance Measures 

Source of data: Ministry of Education 



High levels of student 
achievement 



Reduce gaps in student 
achievement 



High levels of public 
confidence in public 
education 



Performance Measure 



1. Significant increase in the percentage of First Nation, Metis and Inuit students meeting 
provincial standards on province-wide assessments in reading, writing and mathematics 

2. Significant increase in the number of First Nation, Metis and Inuit teaching and non-teaching 
staff in school boards across Ontario 

3. Significant increase in the graduation rate of First Nation, Metis and Inuit students 

4. Significant improvement in First Nation, Metis and Inuit student achievement 

5. Significant improvement in First Nation, Metis and Inuit student self-esteem 

6. Increased collaboration between First Nation education authorities and school boards to 
ensure that First Nation students in First Nation communities receive the preparation they 
need in order to succeed when they make the transition to provincially funded schools 

7. Increased satisfaction among educators in provincially funded schools with respect to 
targeted professional development and resources designed to help them serve First Nation, 
Metis and Inuit students more effectively 

8. Increased participation of First Nation, Metis and Inuit parents in the education of their children 

9. Increased opportunities for knowledge sharing, collaboration and issue resolution among 
Aboriginal communities, First Nation governments, and education authorities, schools, school 
boards and the Ministry of Education 

10. Integration of educational opportunities to significantly improve the knowledge of all 
students and educators in Ontario about the rich cultures and histories of First Nation, 
Metis and Inuit peoples 
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some success. For these programs, the Ministry 
had formulated a strategy, devised a detailed plan, 
and measured results along the way to ensure that 
progress was being made. In contrast, the Ministry 
could not demonstrate how it was systematically 
implementing the Framework. Instead, we were 
informed that most of the Ministry's activities could 
be matched to at least one of the 10 performance 
measures. However, these performance meas- 
ures were developed to assess the success of the 
Aboriginal education strategy and do not contain 
the detail necessary to guide ministry or school 
board activities. 

According to the Framework, Aboriginal edu- 
cation must be integrated into the school board 
planning process. School boards in Ontario must be 
guided by the Board Improvement Plan for Student 
Achievement, a document that helps teachers, 
principals and school board staff plan and imple- 
ment strategies to improve student achievement. 
The Ministry acknowledged that, although 40 
out of Ontario's 72 school boards had included an 
Aboriginal education component in their improve- 
ment plans, it did not have a formal process in place 
to review these or other board plans. The Ministry 
could not provide us with any documentation to 
demonstrate that it had reviewed these board plans 
or assessed the degree to which the Framework has 
been implemented at each school board. 

To assess the Aboriginal education planning 
process at the school board level, we reviewed the 
improvement plans for the past three years at the 
three boards we visited. We found that one of the 
boards made no reference to Aboriginal education, 
while another board listed Aboriginal cultural 
activities and events that it planned for that year. 
The third board set specific targets that focused on 
Aboriginal student achievement, one of which was 
to increase by 10% the number of Aboriginal stu- 
dents who were successful on the Ontario Second- 
ary School Literacy Test (OSSLT). However, none of 
these boards specifically mentioned the Framework 
or identified any of its 10 performance measures in 
their improvement plans. 



An Aboriginal organization engaged by the Min- 
istry in 2009 to obtain feedback on the implementa- 
tion of the Framework had reported that, although 
consistent implementation is necessary to make 
systemic change, implementation of the Framework 
appeared to be dependent on the individual in 
charge of Aboriginal education initiatives at the 
school board. Similarly, based on our visits to the 
three boards, we concluded that there was a need 
for greater ministry co-ordination and guidance. 

Performance Measurement and Reporting 

In 2007, the Ministry committed itself to report 
publicly every three years on the implementation 
of the Framework according to the 10 performance 
measures. It released the first progress report in 
the fall of 2009, titled Sound Foundations for the 
Road Ahead, to provide an update on the progress 
made since 2007 in implementing the Framework. 
However, much of the discussion provided in this 
progress report was of a very general nature and 
did not quantify any of the 10 performance meas- 
ures or identify any specific targets to be used to 
assess progress against these measures. 

For this first progress report, the Ministry 
engaged two Aboriginal organizations to solicit 
community feedback on the Framework, and also 
worked with an independent research team to 
obtain feedback from students, parents and school 
boards. Additionally, the Ministry sent out surveys 
to all school boards. 

We reviewed the reports from the independ- 
ent research team and the two organizations that 
the Ministry used to support its progress report. 
We noted that of the 1,200 students interviewed, 
only 111 had self-identified as Aboriginal students. 
Additionally, almost half the students interviewed 
were from the Ottawa region and 90% were from 
Catholic school boards, which represent only 30% of 
Ontario's elementary and secondary school student 
population. The research team noted that these 
and other sampling concerns limited the amount of 
reliable data collected. Clearly, an unrepresentative 
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or inadequate sample would compromise the integ- 
rity of any related conclusions or inferences in the 
progress report. 

A second progress report on the implementation 
of the Framework was scheduled to be released 
in the fall of 2012. However, in May 2012 the 
Ministry could not provide us with a draft of any 
progress it may have made on the report and stated 
that the report is now scheduled to be released in 
early 2013. 

At the board level, we noted that two school 
boards we visited recently began to assess their 
status with regard to the Framework. This was the 
first time both of these boards had undertaken 
such an assessment since the Framework's release 
in 2007. In October 2011, one of these boards 
prepared a document that identified what it has 
already done and is planning to do to implement 
each of the 10 performance measures. In March 
2012, the other board was asked by the Ministry 
to perform a self-assessment to determine how 
well that board has implemented the 10 perform- 
ance measures. We noted that this process was 
not consistent among all regions and, in general, 
each board operated autonomously in the degree 
to which it implemented specific strategies to 
address the Framework and assessed the success of 
its strategies. 

The Framework's three goals and 10 perform- 
ance measures were developed to enable the 
Ministry and school boards to assess progress made 
toward improving achievement among Aboriginal 
students and closing the gap between Aboriginal 
and non-Aboriginal students. Overall, we found 
that, five years after the release of the Framework, 
neither the Ministry nor the boards we visited had 
measured progress in relation to the Framework's 
performance goals or set any measurable targets 
so that progress could be measured. As a result, 
there is little province-wide information to indicate 
what, if any, progress has been made in improving 
educational success among Aboriginal students and 
closing the achievement gap between them and 
other students in Ontario. 



RECOMMENDATION 1 



To help Aboriginal students succeed in school 
and reduce the gap in student achievement as 
outlined in the Ontario First Nation, Metis and 
Inuit Education Policy Framework (Framework) , 
the Ministry of Education (Ministry) and school 
boards should: 

• develop specific implementation plans 
that identify and address the key obstacles 
faced by Aboriginal students and routinely 
review and update these plans to assess what 
progress is being made; and 

• include in these plans specific goals and per- 
formance measures as outlined in the Frame- 
work and objectively measure and report 
aggregate results to determine whether any 
progress is being made toward improving 
Aboriginal student outcomes. 

| MINISTRY RESPONSE 

The Ministry agrees that implementation plans 
are necessary to support initiatives going for- 
ward to close the achievement gap. Initiatives 
to support Aboriginal student achievement 
must be holistic and leverage evidence-based 
strategies developed in supporting the student 
achievement agenda. 

The Ministry will do the following: 

• Release a preliminary progress report and 
draft implementation plan in fall 2012 

to engage key partners on strategies for 
progress against the Framework's perform- 
ance measures, in order to close the achieve- 
ment gap between First Nation, Metis and 
Inuit students and all students. 

• Release the second progress report on the 
implementation of the Framework in spring 
2013. The report will include an implemen- 
tation plan up to 2016. 

• Align the implementation plan with 
other key government commitments and 
investments that are designed to provide 
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targeted supports for Aboriginal youth, 
such as Ontario's Mental Health and 
Addictions Strategy. 



BOARD RESPONSE 



I 



All three boards agreed with this recommenda- 
tion. One board noted that its process of setting 
specific plans, goals and performance measures 
assists it in measuring student learning over 
time, and that such efforts have resulted in 
improvements for Aboriginal students, par- 
ticularly in primary reading and writing. The 
board noted that it would continue to work on 
further improvements, specifically in the area 
of junior mathematics. A second board's plans 
included strategies to close the achievement 
gap and provide specific supports for Aboriginal 
students. This board stated that it has shared 
student achievement results internally with 
staff and with each band, and that beginning 
in 2012 it would share aggregate data publicly 
in its director's annual report. The third board 
cautioned that the number of students who had 
self-identified was very low for some grades, 
and measuring achievement over time with 
such a small group may not result in reliable 
data. The board noted, however, that such 
information was still useful to support students 
by focusing on tiered intervention and differ- 
entiated instruction. This board also noted that 
annual planning allows it to validate systemic 
learning and, moving forward, it will more 
closely align planning with student learning 
and teacher needs. 



VOLUNTARY, CONFIDENTIAL 
SELF-IDENTIFICATION 

Student Self-identification 

Historically, one of the obstacles related to focusing 
education supports on Aboriginal students is that 



the Ministry and school boards have not been able 
to identify these students in the Ontario school 
system. As a result, data could not be aggregated 
to develop programs, report on results achieved, or 
identify areas for improvement. Consequently, in 
April 2007, the Ministry released Building Bridges 
to Success for First Nation, Metis and Inuit Students 
(Building Bridges) as a guide provided to school 
boards to assist in developing policies for voluntary 
Aboriginal student self -identification whereby stu- 
dents would be given the option to formally identify 
themselves as First Nation, Metis, or Inuit. Board 
implementation of the self -identification policy 
was not mandatory but the Ministry encouraged all 
school boards to develop such a policy. 

The Ministry did not have a formal process in 
place to oversee implementation of self-identification 
policies at the school board level. As of May 2012, 
although 68 of Ontario's 72 school boards had imple- 
mented a self-identification policy, less than half of 
the estimated Aboriginal students had self-identified 
(23,000 out of 52,400) . The Ministry stated that 
the recent implementation of the policy at many 
boards was one of the primary reasons why only 
44% of the estimated number of Aboriginal students 
had self-identified. 

We reviewed data pertaining to self -identification 
at all 72 school boards and identified 18 boards that 
had not shown a significant increase in the percent- 
age of students self-identified over the previous three 
years. In addition, another 30 boards had identified 
less than 20% of their estimated Aboriginal student 
enrolment. In contrast, virtually all the estimated 
Aboriginal students in 16 other school boards had 
self-identified, indicating that it was possible for 
these school boards to successfully implement a 
policy that encourages their Aboriginal students to 
voluntarily identify themselves. 

At two of the three school boards we visited 
virtually all of the estimated Aboriginal student 
population had self-identified. During the first 
year of policy implementation, one of these boards 
had identified 150% of the number of Aboriginal 
students estimated by the Ministry. This board 
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attributed its achievement to creating a success- 
ful awareness campaign prior to implementation. 
Furthermore, this board has received only two 
minor complaints from parents pertaining to stu- 
dent self -identification, which it addressed to the 
parents' satisfaction. At the third board we visited 
only about 50% of the estimated Aboriginal student 
population had self -identified. This board stated 
that it was unaware of the Ministry's estimate of 
the number of Aboriginal students in its schools 
and the Ministry had not provided them with the 
estimate. We believe that boards would find such 
information useful to help them to assess progress 
in identifying their Aboriginal students. 

In July 2011, the Ministry surveyed school 
boards to assess progress to date in the implemen- 
tation of the self-identification policy. More than 
66 boards responded to the survey, and many 
provided recommendations that they thought 
would be useful in increasing the success of this 
policy. The boards wanted to see more ministry 
involvement in the process, with assistance such as 
a public awareness campaign, direct communica- 
tion with Aboriginal communities, and standard 
materials for boards to use. Currently, each board 
independently develops its own materials, such as 
brochures for parents outlining the policy. Some 
boards find this challenging, as they need to ensure 
that information is correct and culturally appropri- 
ate. Boards also recommended that the Ministry 
provide more support in sharing best practices 
province-wide. One of the school boards we visited 
mentioned that boards are struggling to create 
effective strategies and approaches to implement 
the self-identification policy even though they have 
consulted with each other to obtain ideas on how 
to best implement a policy. 

Teaching and Non-teaching 
Staff Self-identification 

The Framework also outlines the need for a sig- 
nificant increase in teaching and non-teaching 
Aboriginal school staff. These individuals can act 



as mentors and role models, share their Aboriginal 
culture, and help foster an environment where 
Aboriginal students can reach a high level of 
achievement. In 2008, the Ministry met with the 
Ontario College of Teachers to develop a self- 
identification process for teachers of Aboriginal 
descent. In 2009, the College reported to the 
Ministry that teachers need to understand what 
self-identification information would be used for 
and how any potential concerns would be miti- 
gated. The College also noted that the successful 
implementation of teacher self-identification may 
take many years and both short- and long-term 
implementation plans are needed. 

At the time of our audit, the Ministry still did not 
have any plans to encourage school boards to imple- 
ment an Aboriginal staff self-identification policy. 
Also, at the time of our audit the Ministry had not 
formally gathered any information on the number 
of boards that had implemented an Aboriginal staff 
self-identification policy or gathered any data on 
the number of staff who have self -identified. 

One of the school boards we visited imple- 
mented a voluntary self -identification policy for 
its Aboriginal staff members in 2008. This board 
has experienced success, as many of its staff who 
self-identified have become resources to reach out 
to Aboriginal students. We noted that the trustees 
at another school board we visited recently passed 
a motion to develop a similar policy. Following our 
discussions with this board, it expressed interest 
in contacting the first board to obtain a copy of its 
staff self-identification policy to help in its own 
policy development. 



RECOMMENDATION 2 



To obtain the population data necessary to bet- 
ter develop specific support programs, report on 
results, and identify opportunities to improve 
Aboriginal student achievement, the Ministry of 
Education (Ministry) should: 
• develop standard communication tools and 
disseminate best practices to assist boards 
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in successfully implementing an effective 
student self-identification process; and 
• develop a policy guide for self-identification 
by Aboriginal teaching and non-teaching 
staff and oversee the effective implementa- 
tion of this policy 

Both the Ministry and school boards should 
exercise effective oversight to help ensure that 
the student self -identification policy is being 
successfully implemented in Ontario schools. 



MINISTRY RESPONSE 



The Ministry agrees that it is necessary to sup- 
port self-identification efforts and commends 
the significant progress that boards have made 
thus far in building trust with Aboriginal fam- 
ilies choosing to participate in the initiative. A 
multi-pronged approach that further engages 
boards and Aboriginal partners is required. 
The Ministry will do the following: 

• Implement a communication strategy to 
support all boards in strengthening their 
Aboriginal student self -identification policy 
implementation by spring 2013. 

• Provide support to all boards in the develop- 
ment of board-specific strategies regarding 
the continued implementation of Aborig- 
inal voluntary self-identification policies 
throughout the 2012/13 school year. 

• Continue to work with education partners, 
including those representing teaching and 
non-teaching staff, to discuss the feasibility 
of the development of a policy guideline for 
voluntary staff self -identification, by the end 
of the 2013 calendar year. 

■ BOARD RESPONSE 



All three boards supported the recommenda- 
tion, with one board noting that sharing best 
practices regarding the implementation of vol- 
untary self -identification would be very helpful, 
especially regarding appropriate and effective 



use of the data. This board noted that the major- 
ity of its eligible students had voluntarily self- 
identified, and this number continues to grow in 
step with the changing demographics of its com- 
munity. A second board stated that it supported 
the development of communication tools that 
can be used as exemplars to augment present 
self-identification processes. The third board 
stated that its self-identification policy had been 
in place since 2007, with almost 100% participa- 
tion of its Aboriginal students. The board noted 
that it had positive results in implementing its 
staff self-identification policy and was willing to 
share its communication documents. 



DATA COLLECTION AND ANALYSIS 

In order to assess progress toward the goals of 
improving student achievement and closing the gap 
between Aboriginal and non-Aboriginal student 
achievement by the year 2016, the Framework 
noted that it will be important to have reliable and 
valid data. In addition, Aboriginal student-specific 
data is needed to support performance measure- 
ment and reporting. To evaluate progress, the 
Framework included three performance measures 
that directly assess Aboriginal student achievement 
and require student-specific data to measure: 

• significant increase in the percentage of First 
Nation, Metis and Inuit students meeting 
provincial standards on province-wide assess- 
ments in reading, writing and mathematics; 

• significant increase in the graduation rate of 
First Nation, Metis and Inuit students; and 

• significant improvement in First Nation, Metis 
and Inuit student achievement. 

The Ministry collects information from schools 
and school boards through a secure, web-enabled 
system called the Ontario School Information Sys- 
tem (OnSIS). The Ministry also acquires data from 
third-party sources such as Statistics Canada and 
the Education Quality and Accountability Office 
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(EQAO), the Ontario government agency that 
performs standardized testing of elementary and 
secondary school students. Once collected, the data 
is run through various quality assurance processes, 
personal information is removed, and the data is 
stored as the authoritative source of education 
data. Our assessment showed that OnSIS is rela- 
tively up to date, as data is submitted, verified and 
finalized three times annually. However, the Min- 
istry does not currently use this data to periodically 
produce standard reports to assess the progress 
made in achieving the goals of the Ontario First 
Nation, Metis and Inuit Education Policy Framework. 

Five years after identifying the challenge of 
closing the achievement gap between Aboriginal 
and non-Aboriginal students, the Ministry has still 
not determined the size of the gap. Further, the 
Ministry has not established a baseline or even a 
starting point from which to measure the gap. The 
Ministry has also not decided when it will begin 
measuring and reporting on Aboriginal student 
achievement, nor has it decided what "significant" 
means in its performance measures, such as to see 
"significant improvement in . . . student achieve- 
ment." Finally, the Ministry has not developed 
specific targets in connection with the performance 
measures as required by the Framework. 

The Ministry stated that it has not begun to 
assess the gap because, in its opinion, the 44% 
of students who have self-identified are not suf- 
ficiently representative to compare their achieve- 
ment results with the results for the non-Aboriginal 
student population. However, the Ministry has 



still not determined the number of self-identified 
students it considers to be enough before it will 
begin measuring and reporting on Aboriginal 
student achievement. 

The Ministry has also not analyzed Aboriginal 
student achievement data for the province as a 
whole, with the exception of some preliminary 
analysis in 2011 that focused on six school boards. 
Therefore, we requested that the Ministry provide 
us with data including EQAO results as well as 
Grade 9 and 10 credit accumulation numbers 
for all self-identified Aboriginal students in the 
provincial school system. Accumulation of credits 
toward graduation is one of the primary indica- 
tors of student success in high school. Using these 
two data sources as a means to analyze student 
achievement, we noted that, for 2011, a significant 
achievement gap continued to exist between self- 
identified Aboriginal students and the rest of the 
student population. We also noted that this gap 
tends to increase as Aboriginal students make their 
way through the educational system, as shown in 
Figure 3. 

The Ministry annually reports the graduation 
rate for the entire student population, but it does 
not calculate a graduation rate for Aboriginal 
students. The Ministry advised us that it will begin 
to report on the graduation rate for Aboriginal 
students following the 2013/14 school year. The 
most recent information available, the 2006 census, 
identified that 65.5% of Aboriginal adults aged 
20-24 had graduated from high school compared to 
88.4% of the non- Aboriginal population of the same 



Figure 3: Comparison of Student Achievement Data, 2010/11 (%) 

Source of data: Ministry of Education 
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age — a gap of 23%, as shown in Figure 1. Our analy- 
sis of the credit accumulation data for 2011 shows 
that only 45% of the Grade 10 self -identified Aborig- 
inal students were on track to graduate from high 
school, compared to 74% of all other students — a 
gap of 29%. These statistics, while not conclusive, 
would certainly indicate that the Ministry faces sig- 
nificant challenges in meeting its 2005 stated goal of 
closing the gap in achievement between Aboriginal 
and non-Aboriginal students by 2016. 

One board we visited analyzed on a regular 
basis indicators such as Grade 9 pass rates, and 
Grade 9 and 10 credit accumulation numbers. 
From this type of analysis, the board identified 
that Aboriginal students were having difficulty 
passing Grade 9 physical education. As a result, it 
introduced culturally relevant games and activities. 
This example shows that analyzing good data 
can uncover previously unsuspected barriers to 
achievement. We also noted that this board sets 
measurable targets specific to Aboriginal student 
achievement. For example, in the 2011/12 school 
year the board aims to increase the percentage of 
self-identified Aboriginal students who achieve 
12 or more credits by the end of the first Grade 10 
semester from 26% to 40%. 

We also researched data collection and analysis 
at various other school boards in Ontario and noted 
that one of the larger boards found that 80% of its 
Aboriginal students did not meet the EQAO prov- 
incial standard on the Grade 6 reading, writing or 
math assessments and, in general, Aboriginal stu- 
dents were highly at risk academically. This board 
reported that it used such information to assess the 
gaps in student achievement and better target its 
support programs. 

Our review of public reporting on Aboriginal 
student results in other jurisdictions revealed that 
the two provinces with the second and third largest 
Aboriginal populations behind Ontario report on 
Aboriginal student achievement results annually. 
British Columbia's Ministry of Education publishes 
an annual report that provides information about 
Aboriginal student performance at both the prov- 



incial and school district levels. Alberta Education 
reports on academic performance measures for 
Aboriginal students directly within its annual 
report. These measures include indicators similar to 
those contemplated in the Framework, such as the 
level of achievement on province-wide standard- 
ized testing and high school graduation rates. 



RECOMMENDATION 3 



To help assess the progress being made toward 
achieving the goals and performance measures 
outlined in the Ontario First Nation, Metis and 
Inuit Education Policy Framework, the Ministry of 
Education (Ministry) and school boards should: 

• establish a baseline with respect to the goals 
and performance measures identified in the 
Framework and set measurable, realistic 
targets; and 

• periodically review progress made with 
regard to closing the gap between Aboriginal 
and non-Aboriginal student achievement so 
that additional or alternative strategies can 
be implemented where necessary. 




The Ministry agrees that it is essential to measure 
and track progress being made toward achieving 
the goals and performance measures outlined in 
the Ontario First Nation, Metis and Inuit Educa- 
tion Policy Framework. The Ministry's approach 
is consistent with lessons learned in its successful 
student achievement agenda. The Ministry now 
has a larger and more representative subset of 
Aboriginal student self-identification data. For 
the next phase of reporting in 2013 and moving 
toward the target of closing the gap by 2016, the 
Ministry will engage with Aboriginal commun- 
ities to develop a reporting approach that best fits 
the current Ontario context. 

The Ministry will do the following: 
• Engage with key First Nation, Metis and Inuit 
partners in fall 2012 regarding its approach 
to public reporting of the Aboriginal student 
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self-identification data in the second 
progress report. 

Release baseline voluntary Aboriginal stu- 
dent achievement data in spring 2013. 
Review First Nation, Metis and Inuit student 
achievement progress annually. 
Continue to support boards and engage 
Aboriginal partners in developing targeted 
strategies to support Aboriginal learners. 
Share best practices and refine implemen- 
tation processes with boards in the 2013 
progress report. 

ARD RESPONSE 

The boards agreed that this recommendation 
was very important, with one board stating 
that data collection and analysis are critical to 
improving achievement, life chances and life 
choices for Aboriginal students. This board went 
on to note that although individual boards can 
use their own data to compare their Aboriginal 
student results over time, they have limited 
access to provincial results for comparison pur- 
poses. A second board stated that it had already 
implemented additional, alternative strategies 
to support Aboriginal students and that a review 
of its data since 2008 demonstrated a continual 
increase in achievement for Aboriginal students. 
The third board stated that while data collection 
and analysis would be very important, the small 
sample size that it gathers would make the cor- 
relation between the impact of board initiatives 
and student achievement unreliable. 



FUNDING 

Since the inception of the Ontario First Nation, 
Metis and Inuit Education Policy Framework in 2007, 
the Ministry has allocated $170 million, primarily 
to school boards, for Aboriginal programs over 
and above regular school board funding. Almost 
$129 million has been provided through the First 



Nation, Metis and Inuit Education Supplement, 
and the remaining $41 million was for Framework 
implementation funding, as detailed in Figure 4. 

First Nation, Metis and Inuit 
Education Supplement 

In 2007, the Ministry introduced the First Nation, 
Metis and Inuit Education Supplement to support 
the goal of improved Aboriginal student achieve- 
ment as outlined in the Framework. This funding 
(per-pupil amount) is intended to help boards 
offer and expand Native Language and Native 
Studies courses and support programs to assist 
Aboriginal students. 

The goal of Native Language courses is to 
instill Native students with pride in their ancestral 
language and to motivate them to use it to com- 
municate in their daily lives. Native Studies courses 
are aimed at providing all students with a broad 
range of knowledge relating to Aboriginal history 
and cultures, and are offered at the secondary 
school level to both Aboriginal and non-Aboriginal 
students. We found that the number of students 
enrolled in both Native Language and Native Stud- 
ies courses has increased since the supplement was 
introduced in 2007. 

We reviewed the per-pupil amount, which is 
the largest component of the First Nation, Metis 

Figure 4: Total Ministry Funding for Aboriginal 
Education Programs, 2006/07-2011/12 ($ million) 

Source of data: Ministry of Education 
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and limit Education Supplement. The per-pupil 
amount is allocated to school boards based on the 
estimated Aboriginal student population as derived 
from the 2006 Statistics Canada census. One of the 
school boards we visited had identified, through the 
self-identification process, 50% more Aboriginal 
students than the number contained in the Min- 
istry's estimate. This illustrates that the information 
the Ministry is using to estimate the number of 
Aboriginal students at each board may not be up to 
date and may not be reflective of the actual student 
population. Consequently, if funds had been allo- 
cated on the basis of the actual Aboriginal student 
population, this board would have received signifi- 
cantly more per-pupil funding than it received. 

The Ministry intends that school boards will 
target this funding to local needs in support of 
activities linked to the Framework's goals and per- 
formance measures. However, the Ministry has not 
placed any specific stipulations on this funding or 
provided any formal guidance on how the funding 
is to be spent. According to the Ministry, this pro- 
vides boards with the flexibility to determine how 
best to allocate resources. In addition, although 
the Ministry may have discussed program spending 
with board representatives, there is no documented 
evidence of such discussions and there is no formal 
report-back process from the boards of any con- 
firmation that the funds have been used to support 
Aboriginal students. 

All three school boards we visited could pro- 
vide us with a brief explanation of how they had 
spent the per-pupil funds. They informed us that 
the funds were used for purposes such as paying 
salaries for Aboriginal support workers, funding 
professional development for teachers, and provid- 
ing resources for students making the transition 
from on-reserve schools to provincially funded 
schools. However, as is consistent with all ministry 
grants for student needs, boards are not required to 
formally track these expenditures to verify that the 
per-pupil funds were used for the benefit of Aborig- 
inal students. In addition, the Ministry has not 
performed any formal analysis on the $60 million 



in per-pupil funding to determine if this funding has 
had any impact on Aboriginal student achievement. 

Framework Implementation Funding 

Framework implementation funding to school 
boards is provided to support Aboriginal student 
achievement and help boards implement the 
Framework. Initiatives under this program include 
self-identification policy implementation, student 
success projects, professional development for 
teachers and student transition programs. School 
boards must submit proposals to the Ministry for 
projects they plan to undertake and, upon project 
completion, are required to submit a report detail- 
ing project outcomes. 

We reviewed the proposal selection process for a 
sample of eight boards and found that there was lit- 
tle documented evidence of proposal prioritization 
by the Ministry or that a formal scoring system was 
used to rank proposals. Also, we were informed that 
ministry staff discussed proposals with each other, 
but these discussions were not documented and 
justification for rejecting proposals was not appar- 
ent or communicated to the school boards we vis- 
ited. In addition, there was no evidence that funds 
were allocated on the basis of either the Aboriginal 
student population at each board or documented 
need. In fact, of the boards we reviewed, the board 
with the fewest Aboriginal students received more 
funding than the board with the greatest Aboriginal 
student population. On a per-student basis this 
worked out to $347 and $73, respectively. 

We reviewed a number of project report-backs 
to the Ministry and found that the reports did not 
include indicators of success even though such 
indicators had been defined by the boards in their 
proposals. Some of the indicators of success noted 
in proposals were to increase Aboriginal student 
credit accumulation and participation in extra- 
curricular activities. Although the project comple- 
tion reports discussed the perceived overall impact 
of the projects, none of the reports reviewed directly 
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addressed the result indicators noted in the related 
original project proposals. 

We also reviewed the Alternative Secondary 
School Program within Native Friendship Centres, 
which provides individualized and culturally 
appropriate supports to help Aboriginal students 
complete their secondary school diploma if they 
have previously dropped out of school or have 
fallen behind because of life circumstances. The 
program is a working partnership between 11 
friendship centres, which are not-for-profit corpora- 
tions mandated to serve the needs of all Aboriginal 
people, and their nearby district school boards. The 
district school board is required to provide teachers 
and course materials for the program. 

The Ontario Federation of Indian Friendship 
Centres (OFIFC), a provincial organization repre- 
senting the collective interests of friendship centres, 
provides administrative oversight of the program. 
The OFIFC flows approximately $90,000 in ministry 
funding to each friendship centre annually. The 
OFIFC collects and sends to the Ministry financial 
information and program outcome reports. These 
reports demonstrate that the funds were used by 
the centres for the purposes intended and provide 
information on the centres' relevant successes and 
challenges including enrolment and exit informa- 
tion, diplomas and certificates awarded, and credits 
attempted and granted. We reviewed the report- 
backs for the past three years and noted that the 
OFIFC had submitted all the reports as required by 
the Ministry. In addition, a 2009 ministry evalua- 
tion of the program found that there were increases 
in student achievement, credit accumulation and 
school retention rates. 



enrolment data, as this could result in a 
more equitable funding allocation; 

• where funding is allocated in response to 
board proposals, document the underlying 
rationale for the funding and communicate 
to boards the justification for accepting or 
rejecting their proposals; and 

• implement report-back processes not only 
to demonstrate that funds are spent for the 
purposes intended but also to obtain infor- 
mation on the success of different types of 
support programs boards are undertaking. 

MINISTRY RESP0NSI 

The Ministry agrees to consider the review of 
existing funding mechanisms and reporting 
procedures. Other funding sources will con- 
tinue to be leveraged to support Aboriginal 
student achievement. 

The Ministry will do the following: 

• Examine the feasibility of remodelling the 
First Nation, Metis and Inuit Education 
Supplement based on available Aboriginal 
student self-identification data. 

• Work more closely with boards to sup- 
port their efforts to promote Aboriginal 
student self-identification. 

• Ensure that all boards receive feedback on 
their proposals, that enhanced reporting 
processes are put in place by 2013, and that 
impacts on student achievement are con- 
sistently evaluated as part of the proposal 
evaluation and reporting processes. 



RECOMMENDATION 4 



To better ensure that funding is allocated based 
on the needs of Aboriginal students, the Min- 
istry of Education (Ministry) should: 
• consider basing per-pupil funding on more 
current and reliable Aboriginal student 



FIRST NATION STUDENTS LIVING 
ON RESERVES 

First Nation Schools 

The education of First Nation students living on 
reserve is the financial responsibility of the federal 
government. Through Aboriginal Affairs and 
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Northern Development Canada, the federal gov- 
ernment provides funding directly to First Nation 
bands to deliver education programs and services. 
There are 118 band-operated schools on reserves 
in Ontario (80 elementary, 7 secondary and 31 
alternative schools). The Ministry informed us that 
approximately 12,700 Aboriginal students attend 
First Nation schools. 

The Ministry's 2009 progress report, Sound 
Foundations for the Road Ahead, identified that 
there was a significant disparity between the level 
of funding targeted for the on-reserve schools by 
the federal government and ministry funding for 
provincial schools. In 2011, the Chiefs of Ontario 
reported that federal funding was approximately 
65% of per-student provincial funding. Research 
suggests that this difference in funding has resulted 
in schools on reserves being unable to pay competi- 
tive salaries to teachers or purchase adequate, up- 
to-date resources for students. This has resulted in 
on-reserve schools often not being able to provide 
the quality of education found in the provincial 
school system. 

The Education Quality and Accountability Office 
(EQAO) may enter into agreements with First 
Nations to test students enrolled in band schools. In 
the 2010/11 school year, 39 out of 118 First Nation 
schools participated in EQAO provincial standard- 
ized testing. These tests help First Nation commun- 
ities assess whether students in their schools are 
meeting provincial standards. We analyzed the most 
recent EQAO results and found that, for participat- 
ing First Nation schools, there was a gap on average 
of 40% from the provincial average for Grade 3 and 
Grade 6 students in reading, writing and math. 
These results reaffirm a 2004 report by the Auditor 
General of Canada and other studies that suggest 
that these students may be performing at a level 
several grades below that which is appropriate for 
their age. 

The Ministry has identified early learning as cru- 
cial to student success. Even though education for 
students who live on reserves is a federal respon- 
sibility, many of these students eventually transition 



into the provincial education system. We inquired 
if the Ministry has worked with the federal govern- 
ment to promote early learning for First Nation stu- 
dents and were informed that, due to jurisdictional 
matters, the province does not have a mandate 
to implement programs or undertake research or 
assessments with regard to students in on-reserve 
schools. Nevertheless, in the 2012 budget, the 
government stated that Ontario is prepared to work 
together with First Nation communities and the 
federal government to share the province's exper- 
tise in delivering education, provided the federal 
government supplies the necessary funding for on- 
reserve First Nation education. 

Tuition Agreements 

According to the Ministry, there are approximately 
5,700 First Nation students who live on reserves but 
attend provincially funded schools. Many of these 
students attend provincially funded schools because 
the reserve does not have a secondary school. Since 
funding for these students is paid by the federal 
government to the band council, these students are 
not considered to be pupils of the school boards and 
therefore the Ministry does not pay the cost of their 
education. Therefore, boards are required to charge 
band councils tuition fees through negotiated 
tuition agreements. Although the Ministry releases 
annual fee regulations that outline a base tuition 
fee, school boards may charge a fee in excess of 
this amount for services such as special education 
programs or specific supports. Amounts charged in 
excess of the base tuition fee must be agreed upon 
by the school boards and First Nation communities 
as part of the tuition agreement. 

The Ministry does not keep track of the tuition 
agreements in place between the school boards and 
First Nation communities in Ontario. However, the 
Ministry estimates that there are approximately 166 
tuition agreements and that school boards collect 
about $60 million per year from First Nation com- 
munities to educate reserve students in provincially 
funded schools. 
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We identified that there should have been a 
separate tuition agreement in place with each of 
the 25 First Nation communities sending students 
to the three school boards we visited. However, we 
noted that there were only 13 agreements in place; 
four agreements had expired, and there were 12 
instances where no tuition agreements existed. 
Nevertheless, we tested a sample of invoices billed 
by school boards to First Nation communities and 
found that the appropriate base tuition fee was 
charged and paid where no agreements existed 
and, in general, First Nation communities were up 
to date with payments. However, in one case we 
found that the school board was charging in excess 
of the base fee where it had no authority to do so, 
since a negotiated tuition agreement did not exist. 
The First Nation community paid the school board 
the base fee, but the school board continued to bill 
excess fees that totalled $1.3 million over the last 
three years. The representative responsible for edu- 
cation at this First Nation community indicated that 
the inability to agree on the amount being charged 
in excess of the base fee was the primary reason a 
tuition agreement had not been signed. 

There are currently no standard ministry policies 
or procedures in place pertaining to tuition agree- 
ments. The Ministry, the federal government, the 
Chiefs of Ontario and four school board associations 
have been working together since 2009 to develop 
tuition agreement guidelines. Rough drafts of these 
guidelines had been distributed to the school boards 
we visited. However, two of these boards were hav- 
ing problems negotiating tuition agreements. Both 
these boards suggested that if standardized template 
agreements could be developed and agreed upon 
by the Ministry, the federal government and First 
Nation communities, this would greatly facilitate the 
process for local boards. 

Transition of Students to the Provincial 
School System 

First Nation students may experience many challen- 
ges when they transition from a school on a reserve 



to a provincially funded school — typically moving 
from a primary school on the reserve to a provincial 
secondary school. For example, some students 
living in remote northern communities leave their 
families to stay with host families in urban areas 
to attend secondary school. This may cause emo- 
tional, physical, spiritual and psychological chal- 
lenges for the student. As part of the Framework 
implementation funding, the Ministry provides 
funding for transition projects intended to assist 
school boards in providing supports to First Nation 
students who live on reserve but attend provincially 
funded schools. 

In the 2011/12 school year, the Ministry pro- 
vided each board with up to $50,000 for Aboriginal 
student success projects, including transition pro- 
jects. We reviewed the transition projects approved 
over the last three years for the boards we visited 
and noted that they often include assigning a lead 
teacher for First Nation students and setting aside a 
room for these students to meet. One of the school 
boards we visited had a formalized Aboriginal 
transition program that included welcome kits, 
assigning a special teacher to transitioning students, 
and offering Grade 9 and 10 transition courses. 
Overall, we noted that the implementation of transi- 
tion programs was at the discretion of the boards. 

We reviewed provincial supports for transi- 
tions in other jurisdictions and found that Alberta 
and Manitoba had created a number of resources 
to help educators with student transitions. For 
example, Manitoba Education has released a policy 
guide to help school boards identify specific student 
needs and develop transition plans. We noted that 
one of the Ontario school boards we visited was 
drafting a secondary school entrance form for First 
Nation students that includes sections pertaining 
to educational background (such as attendance, 
report card marks, standardized testing results and 
strengths in school), as well as any health issues or 
behavioural concerns. 
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First Nation Student Self-identification 

In the 2007 Building Bridges policy guide for 
student self-identification, the Ministry identified 
four groups under which Aboriginal students could 
identify: First Nation (off -reserve), First Nation- 
Tuition Paying (on-reserve), Metis and Inuit. How- 
ever, in October 2011, the Ministry eliminated the 
distinction between on- and off -reserve First Nation 
students. The Ministry explained that it had elimin- 
ated this distinction because some students were 
experiencing self-esteem issues in self-identifying, 
some parents did not understand the distinction, 
and school staff found it difficult to have students 
change their identification when they moved 
on and off the reserve. The Ministry stated that 
these concerns had been informally reported to it. 
However, the Ministry did not assess more formally 
whether these concerns were representative of the 
First Nation community as a whole or attempt to 
alleviate potential concerns before eliminating this 
distinction. The Ministry advised us that the data is 
still maintained internally and therefore its ability 
to target programs to meet the needs of all Aborig- 
inal students would not be impaired. 

On-reserve First Nation students experience 
unique challenges as they transition to schools 
in the provincial system. With a significantly 
greater achievement gap, we believe students 
living on reserve continue to need to be identi- 
fied separately so the Ministry, school boards, 
schools and First Nation communities can target 
specific supports to meet their needs. One of the 
school boards we visited during our audit was 
reviewing its self-identification policy and planned 
to include the First Nation on-reserve category. 
This board had determined that student supports 
can be better targeted if the board collects and 
analyzes student achievement data for on-reserve 
students separately. 

Even though the Ministry does not analyze stu- 
dent achievement data with respect to on-reserve 
students, we found that these students continue 
to have difficulty transitioning into provincially 



funded high schools. For example, in the 2010/11 
school year only half of on-reserve First Nation stu- 
dents attending provincial schools passed the Grade 
10 Ontario Secondary School Literacy Test, com- 
pared to the provincial average of 83%. This indi- 
cates that all stakeholders, including the Ministry, 
school boards, the federal government and First 
Nation communities, need to work more effectively 
to ensure that the needs of these students are 
identified, as more targeted support programs may 
be required to ensure that they are given an equal 
opportunity to succeed. 

Tripartite Agreements 

Since 2009, the Ministry has been engaged in 
discussions with the federal government and First 
Nation communities to formalize education agree- 
ments designed to promote collaboration between 
all parties on initiatives to improve First Nation 
student outcomes. One of these agreements, the 
Education Partnership Program, is a federally led 
initiative where First Nation communities submit 
proposals to the federal government to help 
advance First Nation achievement in both First 
Nation and provincially funded schools. The pro- 
gram supports partnerships between First Nation 
and provincial officials who share expertise and 
services, and co-ordinate learning initiatives. 

Joint initiatives with the federal government 
and First Nation communities to improve the 
academic achievement of students living in First 
Nation communities will not be implemented until 
the agreements have been finalized. However, as 
of April 2012, the education agreements that the 
Ministry has been discussing since 2009 with the 
federal government and First Nation communities 
had not been finalized. 

Eight other provinces, including British Colum- 
bia, have signed tripartite education agreements 
with the federal government and their First Nation 
communities. British Columbia has had an agree- 
ment in place since 1999 and reports annually on 
Aboriginal education results. It reports that over the 
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last five years (2006/07-2010/11) it has reduced 
the gap in high school graduation rates between 
Aboriginal and non-Aboriginal students by 6%. 



RECOMMENDATION 5 



In order to improve educational outcomes for 
First Nation students living on reserves, the 
Ministry of Education (Ministry) and, where 
applicable, school boards, should: 

• develop standardized template tuition agree- 
ments and guidelines that can be used by 

all boards and periodically monitor whether 
valid tuition agreements are in place with 
all bands; 

• take a more proactive role to encourage 
boards to share best practices to assist with 
the transition of students from on-reserve 
schools to the provincial education system; 

• separately measure the effectiveness of 
initiatives implemented to address the 
unique challenges faced by on-reserve 
students attending provincially funded 
schools; and 

• continue to participate in and more pro- 
actively engage in tripartite agreement 
discussions with the federal government and 
First Nation organizations. 

| MINISTRY RESPONSE 

The Ministry agrees that support for First Nation 
students is essential to close the achievement 
gap. The Ministry recognizes that the treaty 
and Aboriginal rights of First Nations people 
are recognized and affirmed in Section 35 of 
the Constitution Act, 1982, and the treaties are a 
fundamental part of the relationship among the 
First Nations, Canada and Ontario. Therefore, 
the Ministry will develop its approach to any 
reporting that may reflect on on-reserve schools 
in partnership with First Nation communities. 
The Ministry will do the following: 

• Release the tuition agreement resource 
guide by the end of the calendar year 




2012. Included in the guide are sug- 
gested examples of key provisions to be 
included in any agreement. The Ministry 
will continue to work with First Nation 
and district school boards to support their 
negotiation processes. 

• Identify and share best practices in the 
development and implementation of tar- 
geted initiatives to support First Nation, 
Metis and Inuit student achievement, includ- 
ing transition programs, in spring 2013. 

• Create and implement successful evidence- 
based targeted programs, based on 
proven success. 

• Develop and enhance successful transi- 
tion programs for students moving from 
schools on reserve to provincially funded 
schools in partnership with boards and First 
Nation communities. 

• Report on provincially funded activities 
undertaken by boards and the Ministry to 
support student transitions. 

• Continue to work with First Nations 
and the federal government to finalize 
and implement tripartite agreements as 
negotiations proceed. 



0ARD RESPONSE 



All three boards recognized the importance of 
helping students with the transition from reserve 
schools to the provincial school system. One 
board noted that it would support any practice 
that encourages integration and the fair treat- 
ment of all students within a strong, accountable 
system. A second board noted that ongoing 
dialogue as well as newsletters and orientation 
sessions helped smooth the transition for its 
students. The third board noted that many of its 
First Nation students leave home, live in a board- 
ing home and move to a large secondary school 
all at the same time. The board stated that, in 
addition to the supports currently in place, it has 
implemented a new program designed to provide 
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academic supports, a safe space with a caring 
adult and a way for students to contact home. 

In regard to tripartite and tuition agree- 
ments, one board noted that expediting efforts 
to finalize the tripartite agreement with the fed- 
eral government would be very helpful to school 
boards. A second board stated that the develop- 
ment of a common tuition template agreement 
may be problematic, since this board serves 
several First Nations with very different tuition 



agreements and formats. The third board stated 
that it agrees with the concept of standardized 
templates for tuition agreements but has found 
that First Nations are reluctant to sign new 
agreements until there is a consensus among 
all First Nations. The board noted that it would 
continue to meet with its First Nation partners 
with the goal of finding mutually acceptable 
agreements that serve their communities and 
students in the best possible manner. 



Chapter 3 
Section 




Ministry of Health and Long-Term Care 

Independent 
Health Facilities 



Background 




Independent health facilities are located in com- 
munities throughout Ontario and provide certain 
health services at no charge to patients insured 
under the provincially funded Ontario Health 
Insurance Plan (OHIP) . About 800 facilities provide 
primarily diagnostic services (such as x-rays, ultra- 
sounds and sleep studies), and about 25 provide 
surgery (such as cataract and plastic surgery) or 
dialysis. Patients generally require a requisition 
signed by their physician in order to receive the 
services, and test results, where applicable, are sent 
to this physician. Under the Independent Health 
Facilities Act, the Ministry of Health and Long-Term 
Care (Ministry) is responsible for licensing, funding 
and co-ordinating quality assurance assessments of 
these facilities. 

The facilities are independently owned and 
operated, with most being for-profit corporations; 
less than 3% are non-profit organizations. The 
Ministry estimates that about half are fully owned 
or controlled by physicians, many of whom are 
radiologists who interpret, for example, x-rays and 
ultrasounds. The Ministry pays facility owners a 
"facility fee" — an amount for each type of service 
provided and/or a contracted amount — for 



overhead costs, such as rent, staffing, supplies 
and equipment. 

In the 2010/11 fiscal year, the Ministry paid 
$408 million in facility fees. Total facility-fee 
payments increased by about 4% per year between 
the 2006/07 and 2010/11 fiscal years, primarily 
because of increased volume of services (because 
the fees paid to facilities for each type of service 
remained largely unchanged) . The total facility-fee 
payments are broken down by type of service in 
Figure 1. 

As well, the Ministry pays physicians working in 
these facilities — often radiologists — a standard fee 
for each service provided, known as a "professional 
fee." The Ministry does not track the total profes- 
sional fees paid to physicians for the specific servi- 
ces they provide in independent health facilities. 

The Ministry contracts with the College of Phys- 
icians and Surgeons of Ontario (the professional 
oversight organization for physicians in Ontario) to 
obtain assurance on the quality of the services pro- 
vided by facilities licensed under the Independent 
Health Facilities Act. As well, the Ministry conducts 
inspections of facilities with x-ray equipment, 
including independent health facilities, under the 
Healing Arts Radiation Protection Act. 
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Figure 1: Total Facility Fees Paid, by Type of Service, 2006/07-2010/11 ($ million) 

Source of data: Ministry of Health and Long-Term Care 





msm 


2007/08 


2008/09 


2009/10 


2010/11 


Diagnostic Services 


Ultrasound 


129.9 


138.0 


150.7 


166.2 


173.0 


Radiology (includes x-rays) 


116.1 


114.6 


110.7 


113.6 


111.0 


Nuclear medicine 1 


39.5 


41.6 


44.0 


42.5 


40.8 


Sleep studies 


29.3 


32.8 


36.5 


39.1 


39.6 


MRI/CT 


5.7 


5.7 


8.4 


8.9 


10.0 


Pulmonary function studies 2 


2.1 


2.0 


2.1 


2.3 


2.4 


Subtotal 


322.6 


334.7 


352.4 


372.6 


376.8 


Surgical/Therapeutic Services 


Dialysis 


9.9 


9.8 


10.2 


11.0 


15.2 


Abortion 


6.4 


6.2 


6.5 


7.0 


7.1 


Ophthalmology 


6.4 


6.5 


6.4 


6.4 


6.4 


Plastic 


1.0 


1.0 


1.0 


1.0 


1.1 


Vascular 3 


0.9 


0.7 


0.7 


0.8 


0.8 


Laser 4 


0.4 


0.3 


0.4 


0.5 


0.4 


Subtotal 


25.0 


24.5 


25.2 


26.7 


31.0 


Total 


347.6 


359.2 


377.6 


399.3 


407.8 



1. Patient tests using radioactive material (e.g., a small amount is administered and images are taken of where it goes in the body to observe blood flow 
through the heart or lungs). 

2. A test to measure a patient's lung function. 

3. Surgery for patients who have problems with their arteries or veins. 

4. Surgery using a laser beam to cut tissue or remove a surface abnormality or lesion. 



Audit Objective and Scop 



— 



Our audit objective was to assess whether the 
Ministry had implemented systems and processes 
to determine whether independent health facilities 
were providing Ontarians with insured services in 
a timely and cost-effective manner, in accordance 
with legislated requirements. Ministry senior man- 
agement reviewed and agreed to our objectives and 
associated audit criteria. 

Our audit work was primarily conducted at 
the Kingston offices of the Ministry's Independent 
Health Facilities Program. As well, we obtained 
information from the Toronto offices of the Min- 
istry's X-ray Inspection Services Unit. In conducting 
our audit, we reviewed relevant documents, 



analyzed information, interviewed appropriate 
ministry staff, and reviewed relevant research 
from Ontario and other jurisdictions. As well, we 
surveyed independent health facilities to determine 
whether certain services were available. We also 
had discussions with the College of Physicians and 
Surgeons of Ontario and the Canadian Nuclear 
Safety Commission related to the operation of 
independent health facilities in Ontario. Further, 
we obtained information and had discussions with 
senior management about similar facilities in other 
Canadian jurisdictions. 

We did not rely on the Ministry's internal audit 
service team to reduce the extent of our audit work, 
because it had not recently conducted any audit 
work on independent health facilities. 



Independent Health Facilities 




Summary 




Our last audit in 2004 found that the Ministry gen- 
erally had adequate procedures in place for ensur- 
ing that independent health facilities complied 
with legislation and ministry policies but noted a 
number of areas requiring ministry action. These 
included determining the service levels necessary 
to meet patient demand, the reasonableness of fees 
paid for facilities' overhead costs, the wait times 
for services and which community-based services 
should be subject to a quality assurance process. 
Since our 2004 audit, the Ministry has taken 
action to enhance the oversight of services offered 
in independent health facilities. For example, it 
has arranged for the College of Physicians and 
Surgeons of Ontario (College) to conduct more 
frequent assessments of facilities with previously 
identified issues; it has also amended the Medicine 
Act, 1991 to enable the College to conduct service- 
quality inspections of community health clinics that 
use anaesthesia (such as those providing plastic 
surgery and colonoscopies) but are not covered by 
the Independent Health Facilities Act. 

Nevertheless, in several other areas the concerns 
we raised in 2004 still remain. For example, min- 
istry data indicates that patients in about 50% of 
Ontario municipalities continue to be underserved 
for certain diagnostic services (including radiology, 
ultrasound and pulmonary function studies) . As 
well, even though the Ministry has not increased 
the rates paid to facilities for overhead costs since 
2006 (in fact, it decreased the rates by 2.5% in 
spring 2012), the Ministry has not researched the 
current actual overhead costs of providing the ser- 
vices. These costs may have changed significantly 
because new technology allowing certain tests to 
be performed much faster often results in lower 
overhead and staffing expenses. 

Ensuring timely availability of services and rea- 
sonable facility fees are both especially important 
because the Ministry's 2012 Action Plan for Health 
Care indicated that a number of less complex 



medical procedures may be moved from hospitals 
into community clinics, such as independent health 
facilities. Some of our other more significant obser- 
vations include the following: 

• Every facility is paid the same amount for 
each type of service provided, regardless 
of the total number of services the facility 
provides. Consequently, larger facilities in 
urban areas often benefit from economies of 
scale, because certain costs (such as rent and 
salaries for reception staff) generally do not 
increase proportionately with the number of 
services performed. Paying slightly higher 
fees in locations with smaller populations 
and lower fees in high-density locations, for 
example, might encourage more facilities 

to service areas that are currently under- 
served without affecting the Ministry's total 
facility-fee payments. 

• The Ministry generally does not allow facilities 
to easily relocate to more underserved areas. 

• The Ministry estimated that certain servi- 
ces — such as MRIs, dialysis and colonoscop- 
ies — were about 20% to 40% less expensive 
if delivered in community clinics, including 
independent health facilities, rather than 
in hospitals. 

• Although the Ministry has some information 
on facility ownership and estimates that about 
50% of facilities are owned or controlled by 
physicians, many of whom are radiologists, 

it has not analyzed the patterns of physicians 
referring patients to their own facilities or 
related persons' facilities. Further, many 
patients assume they must go to a facility 
listed on their physician's referral form, when 
in fact they can choose a hospital or any facil- 
ity that offers the required service. In 2009, 
the Canadian Association of Radiologists 
noted that as many as 30% of CT scans and 
other imaging procedures across Canada con- 
tribute no useful information or are inappro- 
priate. The Ministry estimated that about 20% 
of facility-fee tests are likely inappropriate. 
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• Unlike hospitals, facilities are assessed by the 
College to help ensure, among other things, 
that diagnostic images are correctly "read" by 
the physician, who is often a radiologist. How- 
ever, as of March 2012, about 12% of facilities 
had not been assessed within the last five 
years. Even for the assessed facilities, the Col- 
lege assessors did not review the work of all 
physicians working at each assessed facility. 

• As of March 2012, the Ministry's X-ray 
Inspection Services Unit (Unit) had not 
inspected almost 60% of facilities as fre- 
quently as required to ensure that radiation- 
producing equipment — for example, x-ray 
equipment — was appropriately shielded 

to prevent staff and patients from being 
exposed to excessive radiation levels. 

• The Unit and the Independent Health 
Facilities Program (Program) areas did not 
regularly share information. For example, the 
Unit did not have the current location of 12 
facilities that had moved, so the Unit had not 
inspected whether the radiation-producing 
equipment at the new locations for these 
facilities was safely installed. Further, the 
Unit's inspection reports on facilities were not 
routinely forwarded to the Program. 

• Although the Ministry has attempted to 
improve patient service by introducing two 
websites listing, among other things, certain 
locations where patients can obtain diagnostic 
services such as x-rays and ultrasounds, 
neither site lists all locations offering these 
services. One of the websites, which lists all 
independent health facility locations and ser- 
vices, could be made more user friendly: 

• if it had search capability (for example, 
by postal code or service) to help patients 
locate facilities; and 

• if it included information on facility wait 
times for those services that historically do 
not have same-day access (such as MRIs 
and CTs) to help patients who want their 
tests as soon as possible. 



Detailed Audit Observations 



ACTIONS SINCE OUR 2004 AUDIT 

Since our last audit in 2004, the Ministry has 
undertaken several initiatives related to independ- 
ent health facilities, including the following: 

• In 2006, the Ministry, in conjunction with 
the College of Physicians and Surgeons of 
Ontario (College), initiated unannounced 
reassessments of some facilities and in 2009 
initiated more frequent facility assessments 
for facilities with previously identified 
significant issues. 

• In 2008, the Ministry began receiving from 
the College the names of suspended phys- 
icians who had worked in facilities. 

• In 2011, the Ministry commenced a review of 
independent health facility billings to identify 
questionable billing practices. This review was 
ongoing at the time of our audit. 

ACCESS TO SERVICES 

Distribution of Facility Services 

Before the introduction of the Independent Health 
Facilities Act (Act) in 1990, organizations could 
offer health services outside of hospitals and charge 
patients a fee to cover their overhead costs. When 
the Act became effective, any organizations that 
were already providing the health services covered 
under the Act were "grandparented" if they sent 
a facility application to the Ministry and passed a 
quality assurance assessment by the College. As a 
result, these organizations were permitted to be 
licensed for the services they were providing, in the 
location where they were providing them at that 
time. Once licensed, they could bill the Ministry for 
their facility-fee costs, but were no longer permitted 
to bill patients for their overhead costs. 
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Assessing Service Levels 

Since grandparenting those facilities in existence 
at the time the Act became applicable to them, the 
Ministry has approved only six new licences for 
facilities that bill the Ministry on a fee-for-service 
basis. However, facilities with licences may apply at 
any time to provide services they are not currently 
licensed to provide, in five service categories (radi- 
ology, nuclear medicine, ultrasound, pulmonary 
function studies and sleep studies) or their sub- 
specialties, including mammography (a radiology 
subspecialty) and obstetrical and gynecological 
ultrasound (an ultrasound subspecialty). The Min- 
istry will approve the request only if the area where 
the facility is currently operating is determined 
to be underserved for those diagnostic tests or 
procedures. The Ministry has divided the province 
into a total of 105 areas, primarily municipalities of 
varying sizes. 

In response to a recommendation in our 2004 
Annual Report, the Ministry indicated that its Diag- 
nostic Services Committee (with members from the 
Ministry, the Ontario Medical Association and the 
Ontario Hospital Association) was expected, among 
other things, to make recommendations concerning 
patient access to diagnostic services in underserved 
areas. This Committee was discontinued in 2008, 
and according to the Ministry, no such recommen- 
dations were ever made. 



To assist in determining whether an area is 
underserved or overserved, the Ministry calculates 
the total number of services billed per capita by 
hospitals for outpatients and by independent health 
facilities in the Ministry-defined areas, and com- 
pares these to the provincial average. This calcula- 
tion is performed for the five services as well as for 
more than 15 subspecialties within those services. 
As shown in Figure 2, both hospitals and independ- 
ent health facilities perform a significant number of 
these services. 

The Ministry does not have a benchmark for 
what constitutes a reasonable level of per capita 
diagnostic services. Therefore, the Ministry has 
defined an underserved area as any area providing 
less than 70% of the provincial average per capita 
service level. An overserved area is defined as any 
area providing over 150% of the average per capita 
service level. The Ministry determines all other 
areas (that is, those providing between 70% and 
150% of the average per capita service level) to be 
adequately served. The Ministry indicated that, 
based on its analysis, overall there is an adequate 
supply of the five main services, with about 1,300 
services per 1,000 people in Ontario. 

But the Ministry has not, for the most part, 
analyzed the distribution of underserved and 
overserved areas across the province or over time. 
We analyzed the Ministry's data and noted that, 



Figure 2: Number of Selected Diagnostic Services Performed by Hospitals and Facilities, 2010/11 

Source of data: Ministry of Health and Long-Term Care 





Hospital 








Outpatient 


Facility 




Type of Service 


Services 


Services 


Total 


Radiology (includes x-rays) 


4,515,000 


3,878,000 


8,393,000 


Ultrasound 


2,152,000 


4,267,000 


6,419,000 


Nuclear medicine 1 


752,000 


432,000 


1,184,000 


Pulmonary function studies 2 


593,000 


152,000 


745,000 


Sleep studies 


43,000 


106,000 


149,000 


Total 


8,055,000 


8,835,000 


16,890,000 



1. Patient tests using radioactive material (e.g., a small amount is administered and images are taken of where it 
goes in the body to observe blood flow through the heart or lungs). 

2. A test to measure a patient's lung function. 
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according to the Ministry's definition, about 50% of 
Ontario municipalities (both rural and urban areas) 
had been consistently underserved and about 7% 
had been consistently overserved from the 2007/08 
fiscal year to the 2010/11 fiscal year. We also noted 
that in some cases, an underserved area was next to 
an overserved one. For example, one underserved 
area with 632 services per 1,000 people in the 
2010/11 fiscal year was next to an overserved area 
in which 3,299 services per 1,000 people were 
provided. The Ministry had not analyzed which 
areas it defined as underserved might in fact have 
adequate access to services that could be as close 
as a few kilometres away in neighbouring areas. 
Furthermore, the Ministry's most recent analysis 
of service availability per Local Health Integration 
Network (LHIN, the corporation mandated to work 
with local health providers to determine the health 
service priorities for its region, with the province 
being divided into 14 LHIN regions) was conducted 
in 2007. At that time, the number of services ranged 
from about 1,100 services per 1,000 people in one 
LHIN to almost 3,400 services per 1,000 people 
in another LHIN. Currently LHINs are responsible 
for planning and funding diagnostic services only 
in hospitals, but the Ministry indicated that it is 
considering the extent to which LHINs should be 
involved with planning and funding diagnostic ser- 
vices at independent health facilities in the future. 

Furthermore, the Ministry does not age- or 
gender-adjust the population statistics used for its 
per capita analyses. Because diagnostic procedures 
tend to be required more frequently as individuals 
age, and some services are more likely to be pro- 
vided to women rather than men, factoring in these 
demographics would result in more meaningful 
results. For example, obstetrical and gynecological 
ultrasounds are performed only on women, and 
bone mineral analysis is much more commonly 
performed on older adults, because bones become 
less dense with age. However, all per capita analy- 
ses conducted by the Ministry were based on total 
populations, and did not consider whether an area's 



population had, for example, a higher percentage of 
women or older adults. 

The Ministry does not encourage facility owners 
to relocate to areas within the province that are 
underserved. Consequently, even though popula- 
tions in certain areas of the province may have 
longer wait times for access to these diagnostic 
services, the Ministry's current method of approv- 
ing new services to be offered by existing facilities 
may not be addressing the underservicing issue. 
The Ministry believes that community hospitals 
may be better able to meet local service demands in 
sparsely populated areas. 

The Ministry indicated that, because independ- 
ent health facilities provide only a small portion of 
other services such as Magnetic Resonance Imaging 
(MRI) and Computed Tomography (CT), its analy- 
sis of the per capita levels of these other services by 
LHIN generally excludes those provided by facili- 
ties. However, we noted that the per capita counts 
of CT and MRI machines would rise by at least 10% 
to 50% if machines performing the services at the 
facilities were included. 

Changing Facility Locations 

In order to ensure service availability across 
Ontario, especially in areas that the Ministry has 
determined are underserved, independent health 
facilities are required to obtain ministry approval 
before relocating their operations. The Ministry 
permits facilities to relocate only within their 
current Ministry-defined area, or to within five 
kilometres of the current location if crossing into 
another area. Further, the Ministry will deny any 
move if a hospital within one kilometre of the pro- 
posed new location objects. As well, the Ministry 
will not approve moves from an overserved area 
to an underserved area if the move violates any 
of these rules. In 2011, the Ministry approved 47 
facility moves. 

We noted that some Ministry-defined areas 
are quite large, whereas other areas are small. 
For example, facilities within the city of Ottawa, 
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which is almost 2,800 square kilometres, have 
many more options for where to deliver services 
as compared to facilities located within the city of 
St. Thomas, which is about 40 square kilometres. 
Facilities located in the city of Toronto can move 
anywhere within it. We also noted that there were 
several donut-shaped areas (generally made up of 
the areas surrounding a city) that facilities could 
move around in, but they generally could not move 
to the city within the centre of the area, and facili- 
ties in the centre generally could not move to the 
donut-shaped area. 

The Ministry indicated that many facility owners 
would like to relocate from less populated areas to 
more population-dense locations. Consequently, 
certain owners have attempted to "leapfrog" from 
their current location to a more populated location. 
For example, in one case, over a four-year period, a 
facility applied to move seven times, two of which 
were approved and five of which were denied due 
to exceeding both the five-kilometre rule and cross- 
ing an area boundary. While these relocation con- 
trols are undoubtedly necessary, we expected that 
the Ministry would have considered what changes 
could be made to encourage facilities to move from 
overserved to underserved areas. 

Future Independent Health Facility Services 

The Ministry has analyzed the cost of providing 
certain services conducted in hospitals compared 
with the cost of providing those services in the 
community and has determined that money could 
be saved by moving various services, particularly 
less medically complex ones, out of hospitals and 
into community-based clinics, such as independent 
health facilities. Specifically, the Ministry estimated 
that MRIs and dialysis, as well as colonoscopies 
and echocardiograms (ultrasounds of the heart), 
are less expensive — by a range of about 20% to 
40% — if delivered by a community provider rather 
than in a hospital. At the time of our audit, most of 
these services were more frequently delivered in a 
hospital setting than in the community. 



Ontario's Action Plan for Health Care (Plan), 
released by the Ministry in January 2012, notes 
that more routine procedures may be moved in the 
future out of hospitals and into community-based 
clinics, such as independent health facilities. The 
Plan specifically notes that more cataract proced- 
ures will be conducted by an existing independent 
health facility because the facility can do them at a 
lower cost than a hospital can. Although not men- 
tioned in the Plan, the Ministry indicated to us that, 
in the 2013/14 fiscal year, it expects to increase the 
number of MRIs offered in one existing independ- 
ent health facility, also at a lower cost than in 
hospitals. At the time of our audit, the Ministry had 
no other specific plans to move any other services to 
community facilities. 



RECOMMENDATION 1 



To help ensure that Ontarians have timely and 
convenient access to required tests and pro- 
cedures, the Ministry of Health and Long-Term 
Care should: 

• better identify areas within the province 
where the combined levels of services 
offered by hospitals and independent health 
facilities indicate that the area is under- 
served (for example, by analyzing popula- 
tion and gender distribution within each 
area and determining the resulting needs for 
services) ; and 

• develop ways to help address patient needs 
in regions identified as underserved, such 
as offering incentives to encourage facilities 
to provide services in underserved areas or 
reviewing policies that restrict a facility's 
ability to move into underserved areas. 

MINISTRY RESPONSE 

The Ministry is pleased that the Auditor has 
referenced the Minister's Action Plan for Health 
Care, which communicates our commitment 
to ensuring access for all Ontario residents to 
appropriate health-care services, including 
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timely and convenient access to required tests 
and procedures. Through the Action Plan, more 
services will be provided in the community and 
under a variety of service delivery models suited 
to the service type and community health-care 
needs. The Independent Health Facility (IHF) 
model of service delivery provides a strong 
foundation for moving more diagnostic tests 
and procedures into the community. 

To support the move of more services into 
the community and to ensure that services are 
planned and delivered in accordance with local 
and regional population needs, the Ministry is: 

• actively exploring opportunities for joint 
planning between the Local Health Integra- 
tion Networks and IHFs to ensure the right 
mix and distribution of diagnostic tests and 
procedures to meet local health needs and 
to better identify areas within the province 
where the combined levels of services 
offered by hospitals and IHFs indicate that 
the area is underserved; and 

• enhancing the planning relating to the 
volume of services delivered in the com- 
munity and closer to home for patients, 
with consideration to incentive options to 
attract IHF service providers to underserved 
areas, as well as reviewing its IHF reloca- 
tion policies to better facilitate the move- 
ment of diagnostic tests and procedures to 
underserved areas. 



BILLINGS 

Funding Arrangements 

Fee-for-service Facilities 

Fee-for-service independent health facilities (for 
example, x-ray and ultrasound facilities) are paid 
a standard facility fee for each type of service 
performed. These fees were negotiated between 
the Ministry and the Ontario Medical Association 
(OMA, the bargaining organization that represents 



physicians in Ontario) and are intended to reflect 
the approximate overhead cost (including rent, 
staffing, supplies and equipment) of performing 
each service. The rates are set out in the Schedule 
of Facility Fees for Independent Health Facilities. 
Each facility fee has a corresponding professional 
fee that the physician can charge for performing 
the test or interpreting the test results. Professional 
fees, which are also negotiated between the Min- 
istry and the OMA, are set out in the Ontario Health 
Insurance Plan's Schedule of Benefits. Figure 3 lists 
the top five facility fees billed, the volume of each 
service, and the corresponding professional and 
facility fees per service. The Ministry did not have 
information available on the total professional fees 
billed for these services. 

Between 1992 and 2005, the Ministry made 
some across-the-board decreases and one increase 
to the Schedule of Facility Fees for Independ- 
ent Health Facilities. From 2005 on, no changes 
were made until May 2012, when the Ministry 
implemented a 2.5% across-the-board reduction in 
facility fees and a 5% reduction in their associated 
professional fees "in recognition of the latest evi- 
dence, improvements in technology, and changes 
in standards of care." Reductions were also imple- 
mented for certain other service fees, such as those 
for colonoscopies. 

All facilities are paid the same amount for each 
service, no matter where the facility is located 
or how many services it provides. Consequently, 
larger facilities can benefit from economies of scale 
because certain costs, such as rent and the salaries 
of reception staff, do not increase proportionately 
with the number of services performed. 

A 2000 report produced by the Committee on 
Technical Fees (members of which included the 
Ministry, the OMA and the Ontario Hospital Asso- 
ciation) noted that "cost reimbursement should be 
used as the underlying principle for the funding of 
technical components of diagnostic services" and 
acknowledged that most of the facility fees had not 
been set through a rigorous costing process. The 
Committee suspected that with the introduction of 
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Figure 3: Top Five Total Facility Fees Paid, 2010/11, and Associated Fee Levels 

Source of data: Ministry of Health and Long-Term Care 





2010/11 


As of April 


1, 2011 ($) 




Total Facility Fees 




Facility Fee 


Professional Fee 


1 Type of Service 


Paid ($ million) 


Volume 


Paid per Service 


Paid per Service 


Chest radiology (x-rays-2 views) 


20.0 


826,000 


24.18 


11.30 


Pelvic ultrasound-complete (physician present at 
time of service) 


17.7 


355,000 


50.00 


32.30 


Pelvic ultrasound-complete (physician not present 
at time of service) 


16.4 


328,000 


50.00 


24.20 


Sleep study (initial diagnostic) 


15.5 


41,000 


380.25 


128.30 


Ultrasound of vagina or rectum 


14.6 


291,000 


50.00 


24.20 



new technology and equipment (which make some 
services less time-consuming to perform), some 
fees did not accurately reflect the costs, and noted 
that there was a lack of information on the extent 
to which fees deviated from actual costs. Therefore, 
the Committee recommended that the fee schedule 
be reviewed as soon as possible, and suggested that 
an appropriate costing methodology would ideally 
incorporate factors such as the economies of scale 
available to high-volume facilities. 

In our 2004 Annual Report, we recommended 
that the Ministry "objectively determine the current 
cost of providing each type of service and examine 
the relationship between the volume of services 
provided and the costs of providing services." At 
that time, the Ministry indicated that the Diagnos- 
tic Services Committee would be reviewing this 
area. But this Committee was discontinued in 2008, 
without completing any such analyses, making any 
related changes to the fee schedule or doing any 
work to assess the current overhead costs of run- 
ning a facility. 

The Ministry did, however, provide us with a 
2008 Jurisdictional Comparison of Medical Imaging 
Systems prepared by another province involving 
five Canadian jurisdictions, using data from the 
2006/07 fiscal year. The comparison found that, at 
that time, Ontario's facility fees were often lower 
than fees paid by other provinces. However, the 
Ministry has not investigated whether this is still 
the case, using updated data on the other jurisdic- 



tions' fee levels. We noted that both British Colum- 
bia and Alberta provided a range of services, both 
surgical and diagnostic, in community clinics but 
they did not have separate professional and facility- 
cost fees; rather, the entire payment was combined 
in one fee. We compared the combined facility and 
professional fees paid in Ontario for three services 
to the fees paid for the same services in British Col- 
umbia and noted that the Ontario fees were within 
6%, more or less, of the fees paid there. 

In April 2011, the Ministry requested that 
the OMA (representing physicians), the Ontario 
Hospital Association (representing hospitals) and 
the Coalition of Independent Health Facilities 
(representing facilities) submit facility-fee funding 
options to it by October 2011. The Coalition's 
response did not suggest any funding options; 
it focused on structuring future funding discus- 
sions. Neither the OMA nor the Ontario Hospital 
Association submitted options, although both were 
given an extension until March 2012 to do so. The 
Ministry indicated that it did not expect any further 
submissions relating to facility fees from the OMA 
or the Ontario Hospital Association, because the 
renegotiation of the Physicians Services Agreement, 
which sets out compensation for physicians, was 
the key priority in 2012. 

As previously noted, the Ministry will not 
approve new services for a facility if the area is 
overserved for the services in question. But facilities 
can increase the number of services they currently 
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provide at their existing locations by purchasing, 
without ministry approval, additional diagnostic 
equipment — the Ministry's licence with facilities 
requires only that they inform the Ministry within 
30 days of new equipment being purchased. More 
equipment leading to more services being provided 
results in the Ministry paying more facility and 
professional fees. But if the facilities providing the 
services are in an underserved area, the area bene- 
fits from the increased service availability. 

In contrast to the unlimited number of services 
that facilities can receive payment for, laboratories, 
which are also funded on a fee-for-service basis, are 
subject to a provincial maximum funding level for 
all labs across the province as well as a maximum 
that applies to each laboratory. 

Negotiated Contract Facilities 

The Ministry funds 35 facilities, including facilities 
performing MRIs and cataract surgery, through 
negotiated contracts for providing an established 
volume or number of hours of services. If a facil- 
ity performs fewer services than the contracted 
amount, the Ministry reduces the facility's fund- 
ing. Facilities performing more than the contract 
amount do not receive additional ministry funding. 

The contracted dollar amount (based on volume 
and hours) for facilities that were "grandparented" 
when the legislation was introduced was deter- 
mined through negotiations between the Ministry 
and the facilities. Subsequently, in seven cases, the 
Ministry approved MRI and CT services, which had 
not previously been covered under the Act, through 
a competitive process. For other new services, 
such as cataract surgeries, the Ministry negotiated 
directly with known providers, as permitted under 
the Independent Health Facilities Act. In all cases, 
subsequent contracts are negotiated between the 
Ministry and each facility. 

In 2004, the Ministry established an Expert 
Panel to help improve, among other things, access 
to, and the quality and efficiency of, MRI and 
CT services in hospitals and independent health 



facilities. The Panel's 2005 report indicated that 
these services were not being delivered efficiently. 
In the case of MRIs, this was primarily due to the 
Ministry's having allocated a longer-than-necessary 
time for certain types of MRI procedures and pay- 
ing facilities for rerunning MRIs when the facility 
made an error. In the case of CT scans, the report 
indicated that the tests should take 20 minutes 
regardless of which body part is being scanned, 
whereas the contract rates were based on estimated 
times per service that were, on average, more than 
60% longer. The Ministry told us that it did not 
at that time consider the rate reductions for CT 
scans suggested by the Panel to be appropriate for 
facilities, although it had no analysis supporting its 
conclusions. When renegotiating five-year contracts 
between 2007 and 2009 for the five independent 
health facilities performing MRIs, the Ministry 
again did not consider the Panel's recommenda- 
tions. On the other hand, a couple of years after 
these contracts were negotiated, the Ministry did 
approach these MRI facilities regarding a rate 
reduction. While some of them agreed to reduced 
rates, the others refused. The Ministry continued to 
pay these facilities at the 2007 contract rate while 
paying the other facilities the reduced rate. The 
Ministry indicated that it expected to negotiate a 
reduced rate with the other facilities when the con- 
tract is renewed in 2012. 

With most negotiated contracts, the Ministry 
does not periodically confirm that the rates paid are 
reasonable. Although it might not be practical to 
always conduct a competitive process, the Ministry 
could, for example, periodically obtain information 
from other jurisdictions or otherwise periodically 
review costs to help obtain assurance on the rea- 
sonableness of the rates. Because facility fees are 
not being periodically reviewed, there is a risk that 
some services are increasingly expensive to provide, 
but more of a risk that technological improvements 
have resulted in facility fees significantly exceeding 
the actual cost of performing the services. 
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Referrals for Service 

Patients generally require a physician's referral 
in order to obtain services from an independent 
health facility The Health Council of Canada indi- 
cated in its 2010 report Decisions, Decisions: Family 
Doctors as Gatekeepers to Prescription Drugs and 
Diagnostic Imaging in Canada that increased access 
to diagnostic imaging has "allowed for increased 
use of diagnostic imaging, some of which may be 
considered over-use." The report noted research 
indicating the overuse of diagnostic imaging, but 
concluded that it is difficult to say why and by how 
much. In this regard, the Canadian Association of 
Radiologists' website notes that as many as 30% 
of CT scans and other imaging procedures across 
Canada contribute no useful information or are 
inappropriate. Studies from other jurisdictions also 
indicate the incidence of potentially inappropriate 
imaging. On the basis of studies in two other juris- 
dictions, the Ministry estimated that about 20% of 
facility-fee billings in Ontario are likely inappropri- 
ate — for example, due to unnecessary testing. 

In some cases, the physician referring the 
patient is the physician who performs the service, 
or otherwise works at or owns the facility to which 
the patient is being referred. This is called "self- 
referral," because the physician is referring the 
patient to him- or herself. Physician self -referrals 
can improve patient care in certain situations: for 
example, ultrasounds of the eye may be ordered 
and completed by an ophthalmologist to help 
diagnose and treat suspected eye disease earlier. 
However, various studies, including a Journal of 
the American Medical Association article published 
in 2012, have indicated that "evidence continues 
to mount showing that physicians with ownership 
stakes in imaging equipment are more likely to refer 
their patients for imaging tests than physicians who 
send their patients to radiologists for independent 
imaging." The article further noted that "when 
physicians can refer to scanners they own, [and] 
there is no third-party oversight, they might be 
making subconscious decisions to image." 



One condition for maintaining a licence for a 
fee-for-service facility is ministry approval of any 
changes in the facility's controlling ownership; in 
the calendar years 2008 through 2011, more than 
175 such changes were approved. Facilities are 
required to confirm ownership every five years as 
part of their licence renewal. In addition, some 
owners forward documents indicating the date 
on which changes were made in share ownership. 
However, the Ministry does not otherwise peri- 
odically ask facilities to confirm ownership by, for 
example, having owners tick a box on a form and 
sign if ownership has not changed or state chan- 
ges in writing and sign. As a result, the Ministry 
spends extensive administrative effort in tracking 
ownership structures. 

Although the Ministry estimates that about half 
of Ontario's facilities are fully owned or controlled 
by physicians, it has not analyzed patterns of refer- 
rals by these physicians to their own facilities. The 
Ministry also does not have information on whether 
the physicians who refer patients to a given facility 
have a spouse or other direct relative who owns 
part or all of the facility. The Ministry could use a 
form confirming ownership to also periodically con- 
firm whether the owner (s) or any of their immedi- 
ate family are physicians. 

In May 2012, the Ministry proposed to pay only 
50% of the facility and professional fees for services 
where a physician referred a patient to the clinic 
the physician worked at. The Ministry had not, 
however, estimated the impact of this 50% reduc- 
tion on independent health facilities, although 
ministry documents indicate that some facilities 
might be forced to close operations and that there 
was a potential reduction in the quality and safety of 
imaging or testing services because clinics might use 
"shortcuts" to reduce costs in order to remain viable. 
Furthermore, the Ministry generally did not have 
any information on what might be a reasonable 
proportion of self-referral tests and what proportion 
appeared excessive and thereby warranted follow- 
up. In June 2012, in response to concerns raised 
by the Ontario Medical Association, the Ministry 
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announced that it was establishing a panel to review 
physician self -referrals and would wait for the 
panel's recommendations before changing the fees 
related to self -referrals. 

Verifying Billings 

As shown earlier in Figure I, total payments to 
facilities have increased by about 4% a year, from 
$348 million in the 2006/07 fiscal year to $408 mil- 
lion in 2010/11. In particular, facility-fee payments 
for diagnostic services increased by about 4% a 
year, from $323 million to $377 million. However, 
ultrasound services increased by 7% a year, from 
$130 million to $173 million, over this time period. 
More specifically, with 12 ultrasound services, 
both numbers of services performed and payments 
increased by more than 50% between 2006/07 and 
2010/11. For example, the number of ultrasound 
billings under the code "miscellaneous extremities 
charge per limb" increased 130% from 2006/07 
to 2010/11 or over 20% per year on average, from 
a total of $2.9 million to $7.5 million. Facility fees 
have changed very litde, so most of this change was 
due to increases in the volume of services provided. 
Total payments for contracted services (such as 
dialysis and abortions) increased from $31 million 
in 2006/07 to $41 million in 2010/11. The biggest 
increase was due to the Ministry providing one-time 
funding in 2010/11 for the replacement of about 70 
dialysis machines at the facilities offering dialysis ser- 
vices. Although the Ministry reviews service volumes 
and their effect on total costs to some extent for con- 
tracted services, it has not reviewed the reasons for 
changes in the volume of fee-for-service payments. 

The Ministry indicated that historically it has 
informally identified (for example, through com- 
plaints and assessments of facility expansion appli- 
cations) questionable facility billing practices a few 
times a year, resulting in about one referral a year 
to the Ontario Provincial Police. In November 2011, 
the Ministry began a claims integrity project, which 
involved reviewing the data on facility-fee claims 
for the 2010/11 fiscal year. This review identified 



that about 25% of operating facilities had some 
unusual billing patterns. For example, five facilities 
were billing more than 500 and up to 2,200 com- 
bined head and pelvis ultrasounds on the same visit 
for patients, whereas most facilities billed fewer 
than 50 of this ultrasound combination. Accord- 
ing to the Ministry, this diagnostic combination is 
expected to occur only rarely, because few patient 
conditions can be diagnosed by using it. Despite 
this being a good analysis, the Ministry believed 
that many of the unusual billings could be resolved 
through educating facilities on appropriate bill- 
ing practices and indicated that it was developing 
educational materials that would be shared with 
facilities starting in fall 2012. In addition, although 
the Ministry provided us with a list of potential 
additional data analyses that might identify other 
inappropriate billing practices, no plans were in 
place to conduct such analyses. 

The Ministry provides the College of Physicians 
and Surgeons of Ontario with detailed information 
on the services billed by each facility. However, 
the College informed us that although it does 
receive patient-care data from the Ministry, this 
information is not provided to assessors and that 
the assessors' role does not include any claims 
verification processes. Rather, the College indicated 
that assessors are provided with only the number 
of patients who received a particular service within 
a particular time — for example, the previous few 
months. The College's assessors select a sample of 
procedures performed on patients based on infor- 
mation provided by the facility visited to ensure, 
among other things, that each has an appropriately 
authorized requisition from a physician. However, 
the assessors do not test whether the procedures 
billed to the Ministry were actually performed. We 
also noted that the Memorandum of Understanding 
between the Ministry and College did not state that 
assessors were expected to check facility billings to 
the Ministry, either by using data from the Ministry 
or by any other method. 

Facilities are to send the Ministry information on 
when physicians start or stop working at the facility. 
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The Ministry indicated that it will pay only those 
billings from a facility that relate to physicians who 
are listed as working at that facility. Physicians 
sign the form that facilities submit that states the 
physician's start date, but when facilities submit the 
form that states a physician has left, the physician 
generally does not sign the form, nor is a departure 
date indicated. As well, every facility fee billed to 
the Ministry generally should have a correspond- 
ing professional fee billed by the facility physician 
who interpreted the diagnostic test or, in the case 
of surgery, performed the procedure. The Ministry 
indicated that it does not periodically reconcile or 
spot-check the facility fees and the professional fees 
billed by facility owners and physicians, respect- 
ively, to identify discrepancies. 



RECOMMENDATION 2 



To enhance the cost-effective management of 
the Independent Health Facilities Program, the 
Ministry of Health and Long-Term Care should: 

• periodically review the fee it pays to 
independent health facilities (to cover staff- 
ing, equipment and other overhead costs) 
by assessing the actual costs of the services 
and by making periodic comparisons to 
other jurisdictions; 

• consider alternatives for better managing 
the volume of fees chargeable by facili- 
ties in overserved areas, such as requiring 
these facilities to obtain ministry approval 
before increasing capacity by buying 
more equipment; 

• consider requiring facility owners to declare 
all potential conflicts of interest to the Min- 
istry, and periodically review billing data 

to identify facilities with unusual billing 
patterns, including billings resulting from 
unexpectedly high levels of self -referrals of 
patients by physicians who own or work at 
that facility, or who are related to someone 
who owns the facility — and follow up with 
these facilities; and 



• for selected services, periodically verify that 
facilities have billed the Ministry only for 
services provided to patients — for example, 
through matching facility billings to phys- 
ician requisitions or to the associated phys- 
ician's professional fees for the same service. 

MINISTRY RESPONSI 

The Ministry will consider the feasibility of 
conducting an interjurisdictional review and 
a relative-value-to-cost analysis of the tech- 
nical fees that it pays to Independent Health 
Facilities (IHFs). 

The Ministry agrees that information about 
equipment that is purchased by IHF operators 
is of interest. In this regard, the Ministry estab- 
lished a grant program in the 2011/12 fiscal 
year to assist IHFs in the purchase of digital 
equipment and is considering a second year of 
grant funding for this purpose. Furthermore, 
the Ministry is considering capacity planning at 
the local level involving the Local Health Inte- 
gration Networks, as well as possibly requiring 
approval prior to capacity expansion by IHFs 
operating in overserved areas. 

The Ministry is concerned about self-referral 
and ensuring appropriate utilization of diagnos- 
tic and imaging services in Ontario, including 
services rendered in physician-owned facilities. 
The Ministry's Expert Panel on Appropriate 
Utilization of Diagnostic and Imaging Studies is 
expected to make recommendations in fall 2012 
regarding the appropriate utilization of diag- 
nostic and imaging services rendered in vari- 
ous professional settings, including IHFs. The 
recommendations, when received, will be fully 
reviewed with the Ontario Medical Association 
and actions will be formulated to address issues 
relating to self -referral and appropriate utiliza- 
tion in Ontario, including in the IHF sector. As 
well, the Ministry agrees that the level of phys- 
ician ownership or material interest in an IHF 
business operation is of interest and is actively 




2012 Annual Report of the Office of the Auditor General of Ontario 



exploring mechanisms to require that physicians 
report this information. 

In addition, the Ministry is taking further 
action to implement processes to monitor and 
manage utilization through proactive claims- 
data analyses, provider outreach and education, 
clinic audits, and payment recovery processes. 
The clinic audit process would include verifica- 
tion processes to ensure that facilities have 
billed the Ministry only for services they have 
provided to patients. 



PERFORMANCE MONITORING 

College of Physicians and Surgeons of 
Ontario Quality Assurance Assessments 

To help ensure the quality of services, all facilities 
are required under the Act to have a quality adviser, 
who is responsible for advising the owner about the 
facility's quality and standards of services. Facility 
owners who are physicians may appoint themselves 
as the facility's quality adviser. To obtain independ- 
ent assurance on the quality of the services pro- 
vided by facilities, the Ministry has entered into a 
Memorandum of Understanding (MOU) with the 
College of Physicians and Surgeons of Ontario (Col- 
lege), the professional oversight organization for 
physicians in Ontario. Under this agreement, the 
College is to conduct quality assurance assessments 
of the services provided by facilities using "Clinical 
Practice Parameters and Facility Standards" (Stan- 
dards) developed by the College. 

The assessments are conducted by College- 
appointed assessors — usually a team of one phys- 
ician and one technologist who have experience 
in the facility service (s) being assessed. Assessors 
generally spend about a day at a facility and then 
submit a report to the College. This report indicates 
whether the facility met the Standards and provides 
detailed observations to support the conclusion, 
as well as recommendations for improvement if 
needed. Once reviewed by the College, the report is 
forwarded by the College to the Ministry. 



If the facility does not meet the Standards, the 
report is reviewed by the Ministry's medical con- 
sultant, who advises the Ministry on any additional 
actions that should be taken. When requested to 
do so by the Ministry, the College will conduct a 
reassessment, and occasionally focused assess- 
ments to follow up on certain issues. The Ministry 
may act to suspend or revoke licences based on the 
assessment report's recommendations. 

The Ministry paid the College about $1.5 million 
for conducting the quality assurance processes at 
facilities, which included more than 200 facility 
assessments in the 2010/11 fiscal year. At the time 
of our audit, the Ministry and the College were 
negotiating a revised MOU, which was expected 
to require the facilities to pay the College for 
conducting assessments, rather than the Ministry 
paying the College. 

Determining Assessor Independence 

Because assessors must have experience in the area 
that they are reviewing, most of the physicians and 
technologists who perform facility assessments 
for the College regularly work in hospitals and/ 
or independent health facilities. To help prevent 
potential conflicts of interest between an asses- 
sor and the facility that he or she is assessing, the 
College verbally asks assessors if they have any 
potential conflicts (after providing examples of 
what would constitute a conflict, such as being 
related to someone or having previously worked 
with someone at the facility to be assessed). Asses- 
sors are generally not permitted to assess a facility 
with which they have a conflict. However, the Col- 
lege does not require assessors to sign a document 
declaring that they have no conflicts. 

Although the College is responsible for 
appointing facility assessors, because of the com- 
petitiveness of certain facilities, facilities have the 
option to reject an assessor and have an alternate 
appointed in certain situations (such as potential 
conflicts of interest). The College informed us that 
facilities exercise this option about 2% of the time. 
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Scheduling Assessments 

The Ministry uses its database, which includes the 
dates of all facility assessments and reassessments, 
to identify which facilities it will request the Col- 
lege to assess during the next year. The Ministry's 
general policy is to have each facility assessed every 
five years. It provides that year's list to the College, 
which is responsible for scheduling assessments 
with each facility. 

We found that, as of January 2012, the College 
had not assessed within the previous five years 
about 12% of the facilities that should have been 
assessed. The Ministry indicated that assessments 
could be delayed for various reasons, including a 
lack of assessors with service-specific knowledge 
(for example, the College indicated that assessors 
for sleep study clinics are difficult to find because 
fewer people specialize in this area) and facilities 
postponing assessments because of proposed moves 
to other locations. 

In our 2004 Annual Report, we recommended 
that the Ministry consider having the College 
perform at least some assessments without 
advance notice. At the time of our current audit, 
facilities were always provided with at least six 
to eight weeks' notice so that they could prepare 
for the visit. (Reassessments may be unan- 
nounced, as discussed further under Following Up 
on Assessments.) 

Conducting Assessments and Reporting 
Assessment Results 

The College's assessors use a checklist based on the 
College's "Clinical Practice Parameters and Facility 
Standards" (Standards) to conduct their assess- 
ments. The Ministry has a copy of the Standards but 
generally does not receive a copy of the checklists 
used. To complete the checklist, assessors select 
and review a sample of test images (in the case of 
diagnostic facilities) and/or patient charts, surgical 
notes, and so on (in the case of dialysis and surgical 
clinics) ; where possible, they also observe tests 
being performed on patients. We noted that the 
College's sampling guideline recommended using 



the same minimum sample sizes (between 10 and 
15 per type of service provided, depending on the 
type of facility) for all facilities providing a particu- 
lar service, regardless of the number of procedures 
provided by the facility. We also noted that the 
assessors' approach to choosing this sample did not 
ensure that they reviewed the work of all profes- 
sionals at the facility. 

After it receives the assessors' report, the Col- 
lege is to report assessment results to the Ministry 
within three to 20 business days, depending on 
the seriousness of the assessment results (these 
time frames were implemented in 2005) . We noted 
that the College generally met these deadlines in 
the 2010/11 fiscal year. But there is no established 
deadline by which assessors must report results to 
the College. We noted that the time between the 
assessment date and the date that the Ministry 
received the assessment results from the College 
was a median of 47 days in the 2010/11 fiscal 
year, with 90% of reports being received within 84 
days. Most of this longer time frame was the time 
between the assessment date and the College's 
receipt of the report from the assessor. 

The assessment results provided to the Ministry 
rate facilities on a scale of one (the highest rating: 
the facility is following the required standards, and 
there are no recommendations) to five (the lowest 
rating: patient care is at risk, and recommendations 
may include the immediate closure of the facility) . 
Between January 2007 and January 2012, more 
than 80% of the approximately 1,100 facilities that 
were assessed were rated as meeting the standards 
with few or no significant recommendations (that 
is, the facilities received one of the top two ratings). 

Following Up on Assessments 

The College generally determines whether facilities 
require any additional follow-up after an assess- 
ment, based on feedback from its assessors. Follow- 
up can range from requiring the facility to forward 
documents, such as an action plan detailing how it 
will address identified deficiencies, to the Ministry 
requesting the College to conduct a reassessment. 



2012 Annual Report of the Office of the Auditor General of Ontario 



Reassessments are generally conducted three 
months to a year after the original assessment, 
based on the College's judgment. Six facilities were 
reassessed in the 2010/11 fiscal year. 

In our 1996 and 2004 Annual Reports, we 
noted that the Ministry and the College had not 
agreed on timeframes for the College's follow-up 
activities, and recommended that the Ministry 
update its Memorandum of Understanding with the 
College to incorporate such timeframes. However, 
no timeframes were in place at the time of our 
current audit. 

The Ministry can ask the College to conduct its 
planned reassessments on an unannounced basis. 
In these cases, the College either sends assessors 
in completely unannounced or notifies the facility 
that an assessment will be conducted within the 
next month, without providing a specific date. We 
noted one case where the facility denied the asses- 
sors entry on two "unannounced" occasions in 2010 
before providing access in March 2011, even though 
the Independent Health Facilities Act states that all 
facility owners are required to co-operate fully with 
assessors. However, the Act has limited penalties for 
refusal to co-operate, and the College told us that it 
does not have any authority to impose penalties in 
this situation. 

In the 2009/10 fiscal year, to obtain more 
assurance that facilities with previously identi- 
fied and rectified problems were continuing to 
meet standards, the Ministry asked the College to 
assess certain facilities every two to three years, 
rather than the standard five years. In the 2010/11 
fiscal year, the College conducted 26 of these 
mid-cycle assessments. 

Inspections Conducted by the X-ray 
Inspection Services Unit 

While the College's standards require facilities 
to check radiation-producing equipment every 
six months, the Ministry is responsible under the 
Healing Arts Radiation Protection Act (HARP Act) 
for periodically inspecting a facility's radiation- 



producing equipment, including primarily the 
equipment used to perform x-rays but also the 
equipment used for CT scans and fluoroscopy 
(which involves a series of x-rays that produce a 
continuous moving image on a monitor, giving, 
for example, a picture of the movement of contrast 
dye through a patient's body). The Ministry's X-ray 
Inspection Services Unit (Unit) is responsible 
for these inspections, as well as for ensuring 
the appropriate initial installation of radiation- 
producing equipment (for example, sufficient lead 
in the walls for shielding) . The Unit also ensures 
that x-ray equipment does not produce radiation 
in excess of standards set out in the HARP Act. 
Radiation produced by CT equipment is not 
reviewed by the Unit because there are no similar 
standards for CT equipment. 

The Unit is responsible for inspecting just over 
7,600 locations — including hospitals, dental sites 
and more than 450 independent health facili- 
ties — with radiation-producing equipment. In 
the 2011/12 fiscal year, the Unit inspected a total 
of nearly 1,700 locations, including about 70 
independent health facilities. 

The Unit has four inspectors who perform 
announced visits that are scheduled using 
risk-rankings for the various types of facilities. 
For example, independent health facilities are 
required to be assessed within one year of opening 
a new location and every two to three years after 
that, unless they receive a ranking of "bad" as 
a result of an inspection, in which case they are 
inspected annually until the situation is rectified. 
Ministry policy is to inspect all radiation-producing 
equipment at new locations and about 25% of 
the equipment at established locations. The Unit 
cannot easily determine the percentage of facility 
equipment tested because it did not document all 
equipment in use at the inspected facilities. We 
also noted that ministry policy does not indicate 
that inspectors should ensure that all equipment is 
tested over time; rather, the policy indicates that 
higher-risk equipment, such as fluoroscopy equip- 
ment, should be tested. 
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We reviewed the data from the Unit's inspec- 
tion database and found that as of March 2012, 
almost 60% of the independent health facilities 
had not been assessed within the Unit's prescribed 
time frames. Furthermore, the Ministry could 
not determine how many of these facilities were 
new or how many had been rated as "bad" in their 
last inspection. 

The Unit does not rely on the work of other over- 
sight entities, including the Canadian Association 
of Radiologists, the Ontario Association of Radiolo- 
gists and the Ontario Breast Screening Program, 
all of which review certain types of radiation- 
producing diagnostic imaging equipment, including 
equipment in independent health facilities. The 
work of these organizations might enable the Unit 
to reduce its time at some facilities and focus its 
efforts on facilities and/or equipment that are not 
otherwise being tested. 

The Ministry had no policies or procedures on 
what type of information should be exchanged 
between the Unit and the Ministry's Independent 
Health Facilities Program; rather, communications 
occur at the discretion of staff in each ministry 
area. We noted that minimal communication or 
exchange of information took place between the 
Unit and the Program. For instance, the Unit's 
inspection system and the Program's database were 
not linked, and although the Ministry indicated 
that data comparisons to identify discrepancies 
could be conducted manually, this has not been 
done. As a result, for example, the Unit incorrectly 
categorized almost 40 independent health facilities 
as dental offices, which are generally inspected 
every five years rather than every three years for 
independent health facilities. The Unit also had two 
facilities that were actually open incorrectly listed 
as "closed" and did not have current information on 
12 facilities that had moved more than a year previ- 
ously. Furthermore, the Unit's inspection reports 
were not forwarded to the Program — although the 
inspection staff told us that they would inform the 
Program of any significant issues. 



Ministry Monitoring 

Suspensions and Revocations of Licences 

The Independent Health Facilities Act (Act) states 
that the Ministry can revoke, suspend or refuse to 
renew a facility's licence for a variety of reasons. 
Problems can include quality assurance issues 
(such as equipment requiring maintenance) and 
operational issues (such as facilities operating 
out of unapproved locations; ceasing to operate 
for at least six months without taking reasonable 
steps to prepare to reopen; or transferring over 
50% of a facility's ownership without obtaining 
ministry approval) . 

The Act allows the Ministry to immediately 
suspend a facility's licence if there is an immediate 
risk to a patient's health or safety. We noted that 
the last suspension occurred in 2011. The Ministry 
indicated that in most cases, there is no immediate 
risk to patients. In these cases, under the Act, the 
Ministry issues a "proposal to suspend" a facility's 
licence, an approach that provides the facility with 
time to correct any identified quality assurance 
problems in order to avert a licence suspension. 
Between January 2007 and January 2012, the 
Ministry issued such notices to 32 facilities. The 
Ministry ultimately suspended seven for not taking 
the required corrective action on a timely basis. 

The Ministry can revoke the licence of sus- 
pended facilities that do not correct identified 
problems, as well as facilities with operational 
issues. The Ministry's records indicated that six 
licences were revoked from January 2007 through 
January 2012. Most of these revocations occurred 
because the facility had ceased operations for more 
than six months and did not make sufficient efforts 
to reopen. 

When the Ministry proposes to suspend or 
revoke a licence, the Act allows the facility owner 
15 days to request a hearing with the Health 
Services Appeal and Review Board. Facilities that 
request a hearing can continue to provide services 
to patients and bill the Ministry until the Board 
makes a decision. We noted that between January 
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2007 and January 2012, facilities requested hear- 
ings regarding seven proposals to revoke. Of the 
21 hearing requests that were resolved between 
January 2007 and January 2012, 17 were settled 
between the Ministry and the facility owner before 
the Board reached a decision, three were not 
pursued by the facility owner, and the Board made 
a decision on the remaining one. Most of the 17 
that were settled involved three facility owners. 
We also noted that the average time from request 
for hearing until resolution was about five years 
for quality assurance issues and two years for 
operational issues. 

Even though the Act gives the Ministry signifi- 
cant discretion regarding whether or not to license 
facilities if it has reasonable grounds to believe that 
a facility is not being operated in accordance with 
the law or with honesty and integrity, this provision 
is rarely used. For example, the Ministry licensed 
an owner to provide a new service even though 
ministry staff had identified significant concerns 
about this owner's billings the year before (a full 
investigation of the billings irregularities was still 
under way when the owner was licensed for the 
new service) . 

Other Monitoring Activities 

The Ministry is made aware of activities or circum- 
stances that constitute a contravention of the Act in 
a number of ways, including complaints by the pub- 
lic and other facility owners. However, the Ministry 
has no information about complaints made directly 
to facilities. The Ministry tracks the complaints 
it receives (mostly from the public) that indicate 
quality-of-care deficiencies or other violations of 
the Act. About 35 such complaints were tracked in 
2011, with over half pertaining to quality assurance 
issues such as equipment not being disinfected 
between patients and facility staff not behaving 
professionally. The Ministry's complaint follow-up 
activities varied depending on the nature of the 
complaint. For example, in two cases the Ministry 
asked the College of Physicians and Surgeons to 



conduct a focused facility assessment to review 
the issues. 

The Ministry also conducts other administrative 
activities to ensure compliance with the Act, some 
of which may involve time-consuming processes. 
For example, as previously noted, ministry staff 
make extensive efforts to track information on facil- 
ity owners. And every month, ministry staff manu- 
ally review a report on facility billings, track which 
facilities have not billed in the previous six months 
and send a letter to those facilities. The letter asks 
the facility to either confirm that it has ceased oper- 
ations or provide the steps it is taking to reopen. 
A follow-up letter is issued after an additional six 
months if necessary, and if billings do not begin 
again, more progressive actions, such as a proposal 
to revoke the facility's licence, are undertaken. 
Between January 2007 and January 2012, the Min- 
istry sent about 200 letters regarding 100 facilities 
that did not bill for at least six months. 

While these may be worthwhile efforts, we 
expected that the Ministry would have assessed 
whether its time might be more effectively used 
following up on those facilities with unusual 
or possibly inappropriate billing practices, as 
discussed previously. 



RECOMMENDATION 3 



To better ensure that independent health facili- 
ties are providing services according to quality 
medical standards established by the College of 
Physicians and Surgeons of Ontario (College) 
and are meeting other legislated requirements, 
the Ministry should: 

• work with the College to ensure that every 
facility is inspected at least once between each 
five-year licence renewal for that facility; 

• consider including additional expectations in 
its Memorandum of Understanding with the 
College, such as: 

• requiring assessors to review the quality of 
each physician's work at the facility; and 

• requiring that assessment results for 
facilities with significant issues be more 



Independent Health Facilities 




promptly reported to the Ministry after 

the assessment; 
consider, when next reviewing the 
Independent Health Facilities Act, adding 
penalties for facility owners who refuse 
access to the College's assessors when they 
arrive unannounced; 

develop policies and procedures to improve 
information-sharing between the Ministry's 
Independent Health Facilities Program and 
its X-ray Inspection Services Unit, includ- 
ing information on the location of facilities 
offering x-ray services as well as information 
on inspection results, so that each has the 
most current information available on the 
facilities they oversee; and 
consider options for streamlining the 
monitoring of facilities' activities, including 
determining whether the Ministry's X-ray 
Inspection Services Unit can rely on the work 
of other professional or federal oversight 
entities to enable it to focus its activities on 
the newer or higher-risk facilities. 



frequency as other physician specialties. As well, 
with respect to the College's prompt reporting of 
facilities with significant issues to the Ministry, 
a working group has been established to review 
reporting policies and to suggest appropriate 
standards for the turnaround time from the 
date of assessment to notifying the Director of 
significant issues. 

The Ministry will consider implementing 
penalties or other appropriate provisions in 
situations where facility operators deny access 
to the College's assessors. 

Patient safety is a ministry priority. The X-ray 
Inspection Unit is embarking on a review of its 
inspection delivery model and is collaborating 
with other branches of the Ministry in a broader 
review of the Healing Arts Radiation Protection 
Act (HARP Act) to identify options to enhance 
the oversight of x-ray machines in Ontario. 
The Ministry has made a specific commitment 
to review the current HARP Act to identify 
opportunities that would help promote continu- 
ous improvement in the safe use of radiation- 
emitting medical imaging devices. 



The Ministry is working with the College to 
ensure that every independent health facility 
is inspected at least once within the five-year 
renewal cycle. Although there may be excep- 
tional circumstances (for example, where servi- 
ces are not currently being provided or when a 
facility is in the process of relocating, changing 
owners or expanding the scope of services being 
provided), the Ministry will work closely with 
the College to minimize such exceptions. 

The Ministry agrees with the recommen- 
dation with respect to requiring assessors to 
review the quality of each physician's work. At 
this time, the College is in the process of align- 
ing the quality assurance assessment program 
for independent health facilities with its peer 
review program for physicians, to ensure that 
radiologists are peer-reviewed at the same 



COMMUNITY HEALTH-CARE CLINICS 
NOT COVERED BY THE ACT 

Community health-care clinics fall into 
two categories: 

• those that are covered by the Independent 
Health Facilities Act (Act), which we refer to 
throughout this report as "independent health 
facilities" (facilities), and 

• those that are not covered by the Act, which 
we refer to as "community health-care clinics" 
(or community clinics) . 

In our 2004 audit of independent health facili- 
ties, we noted that some diagnostic procedures, 
such as colonoscopies, were not licensed services 
under the Act and were therefore performed at 
community health-care clinics not covered by 
this legislation. We also observed that a licensed 
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service, abortion, was being performed both in 
independent health facilities and in community 
clinics. At that time, neither community clinics pro- 
viding colonoscopies nor community clinics provid- 
ing abortions were subject to the College's quality 
assurance assessments. 

In 2007, the Ministry licensed one independent 
health facility in Northern Ontario to provide, 
among other things, colonoscopies. The Ministry 
had no documented rationale for why only this 
community clinic was licensed as a facility to 
provide colonoscopies. At the time of our cur- 
rent audit, the Ministry had no information on 
the number of non-facility community clinics 
operating in Ontario that were performing colon- 
oscopies. However, information at the Ministry 
indicated that in the 2010/11 fiscal year, about 
36% of all colonoscopies were performed in 
community clinics. 

In 2010, legislative changes to the Medicine Act, 
1991 made the College of Physicians and Surgeons 
of Ontario (College) responsible for inspecting all 
community clinics that use anaesthesia to provide 
insured services (such as cosmetic surgery and 
colonoscopies) and uninsured services (such as 
hair transplants). However, the Ministry does not 
obtain information on the frequency or outcome 
of these inspections. We contacted the College, 
which informed us that since commencing such 
inspections in fall 2010, it had inspected about 
50 community clinics performing colonoscopies 
and four community clinics performing abortions. 
The College indicated that most of the community 
colonoscopy clinics had passed their inspections 
with some conditions. No information was pro- 
vided by the College regarding the community 
abortion clinics. 

We also noted that about 50 of the x-ray sites 
inspected by the Ministry's X-ray Inspection Servi- 
ces Unit were community clinics, not independent 
health facilities. These clinics cannot bill the Min- 
istry or their patients for facility fees for insured 
services, but physicians working in these clinics 
can bill the Ministry for the related professional 



fees. The Ministry's Independent Health Facility 
Program was not aware of these community x-ray 
clinics, and, unlike the case with independent 
health facilities, the x-rays read by radiologists 
in these clinics are not periodically reviewed by 
the College. 



RECOMMENDATION 4 



To ensure that all community clinics providing 
insured services — even those that do not use 
anaesthesia — offer quality medical services, the 
Ministry of Health and Long-Term Care should 
consider engaging the College of Physicians 
and Surgeons to oversee those clinics that 
offer services that would be subject to College 
oversight if they were classified as independent 
health facilities. 



The Ministry agrees that the provision of qual- 
ity health care that includes quality medical 
services is a priority and established Health 
Quality Ontario (HQO) to lead the Ministry's 
quality and evidence-based agenda. Among 
other things, HQO makes recommendations to 
the Ministry regarding the quality of health- 
and medical-care services provided to patients, 
including services provided by community and 
specialized clinics, and also provides guidance 
and makes recommendations to health-care 
providers and relevant organizations on stan- 
dards for patient care based on evidence, and 
clinical best practice guidelines and protocols. 

Furthermore, Cancer Care Ontario and 
the College are working in partnership to plan 
and implement a model for quality improve- 
ment focused on selected health services 
that are delivered in settings other than an 
independent health facility, including pathol- 
ogy, colonoscopy and colposcopy. The Ministry 
expects to receive an implementation plan by 
March 2013. 
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PUBLIC INFORMATION 

Patients have the right to choose which independ- 
ent health facility they go to. To make these 
choices, patients need information on which facili- 
ties provide the service (s) they require and where 
these facilities are located. 

Referral Forms 

Patients requiring services provided by an 
independent health facility often get a referral form 
from their physician that provides contact informa- 
tion for one facility or for several facilities owned 
by the same owners. Many patients assume that 
they must go to a facility on the referral form; but 
in fact they are free to choose a hospital or any facil- 
ity that offers the required service (s), including a 
facility that may be closer to home. As noted earlier, 
some physicians have a financial interest in certain 
facilities, which, as several studies have indicated, 
may influence both the rate at which they prescribe 
certain tests and which facilities they refer patients 
to. In contrast, referral forms for laboratory ser- 
vices (such as blood tests) are standardized and 
do not readily lend themselves to listing specific 
laboratories. At the time of our audit, the Ministry 
did not have any plans to develop a standardized 
referral form for services provided by independent 
health facilities. 

Ministry Website 

The Ministry has a website (the Health Care 
Options Directory) that allows the public to search 
for locations providing services such as x-rays and 
ultrasounds. But this site does not list all of the 
independent health facilities and hospitals provid- 
ing these services. Rather, it is primarily a listing 
of these services provided outside of facilities and 
hospitals, such as in after-hours clinics. In 2010, the 
Ministry introduced another website that lists only 
the independent health facilities in Ontario. The 
list includes each facility's address and telephone 



number, as well as all the services each facility is 
licensed to provide. The Ministry told us that this 
list is updated monthly and that patients interested 
in identifying alternative facilities can review 
this list. 

However, we noted that the list was not as help- 
ful as it could be. For example, unlike the Health 
Care Options Directory website, this website had 
no tools allowing users to search by postal code or 
by service area. More significantly, the list did not 
always offer an accurate description of the services 
offered at a given facility: a clinic listed as licensed 
to offer a specific service might not actually offer 
that service. When we contacted a sample of facili- 
ties that the website indicated were licensed to 
provide fluoroscopy services, 35% said that they 
did not provide such services at all, and another 
20% said that they had temporarily suspended the 
service because of machine-maintenance or other 
issues. One facility's phone number was no longer 
in service, and a staff person at another directed 
us to a different location, which we noted was not 
licensed to provide the service. (The Ministry was 
following up with this facility.) The website also 
did not list the specific services provided at a given 
facility. For example, not all facilities offering ultra- 
sound services perform knee ultrasounds. 

We noted that as of January 2012, the list 
included addresses for almost 950 facilities; how- 
ever, information at the Ministry indicated that 
there were about 800 unique facility locations. We 
were informed that some facility owners operate 
with multiple licences out of one location. As well, 
we noted that more than 20 listed facilities had not 
been open for at least five years. 

In our 2004 Annual Report, we recommended 
that the Ministry consider publicly disclosing any 
serious quality assurance problems at independ- 
ent health facilities. We noted during our current 
audit that the Ministry's website included the dates 
and results of each facility's last quality assurance 
assessment. However, it did not indicate which 
facilities had been suspended for failing to meet 
required standards; instead, suspended facilities 
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were just removed from the online list. Because sus- 
pended facilities may continue to offer patient ser- 
vices (they cannot bill facility fees, but can still bill 
professional fees), the Ministry requires suspended 
facilities to remove their licences from their walls 
so that patients know that they are suspended. But 
the Ministry has not analyzed whether this is an 
effective way of informing patients that the facility's 
licence is suspended. 

The website also provides no information 
about the process for filing a complaint about an 
independent health facility. 

Wait Times 

The Ministry's Wait Time Strategy website reports 
wait times for MRI and CT examinations at 
hospitals, and for cataract surgeries at hospitals 
and at one of the two independent health facili- 
ties providing these surgeries. The Ministry does 
not report the wait times for an MRI or CT at the 
seven independent health facilities offering these 
services. As a result, patients cannot "shop" for the 
shortest wait time for these services. As well, the 
Ministry does not collect or report wait times for 
other diagnostic services provided by independent 
health facilities, such as x-rays and ultrasounds, 
because the Ministry does not expect significant 
waits for these services. Although we would 
not expect the Ministry to track services with 
same-day access (such as many types of x-rays), 
public information on wait times for services 
that have historically involved a significant wait 
(such as MRIs, CTs and cataract surgeries) might 
be beneficial. 



RECOMMENDATION 5 



To ensure that patients have access to relevant 
information about independent health facilities 
that can help them obtain required services, the 
Ministry of Health and Long-Term Care should: 

• consider the costs and benefits of introdu- 
cing a standardized referral form, similar to 
that used in the laboratory program, that 
restricts physicians from recommending a 
preferred facility and that contains informa- 
tion about how to locate an independent 
health facility using the Ministry's website; 

• combine existing website information into one 
website with search functionality that speci- 
fies all locations where patients can access 
community services, such as x-rays and ultra- 
sounds, as well as available services and wait 
times for services that do not have same-day 
access (for example, MRIs and CT scans); and 

• provide information on its website regarding 
how to register a complaint about an 
independent health facility. 



| MINISTRY RESPONSE 




The Ministry agrees that public information- 
sharing accomplished through accessible and 
user-friendly formats, tools and mechanisms is 
important. In this regard, the Ministry will: 

• explore options for standardizing diagnostic 
referral forms, including the potential for 
electronic referral formats; 

• explore options to combine existing website 
information to ensure patients have access 
to comprehensive information (the Ministry 
is in the process of having independent- 
health-facility MRI and CT service wait times 
reported through the Provincial Wait Times 
Strategy) ; and 

• include information on its website on how to 
register a complaint about an independent 
health facility. 



Chapter 3 
Section 




Legislative Estimates 
Review Process 



Background 




GOVERNMENT SPENDING AND THE ROLE 
OFTHE LEGISLATURE 

In the Canadian parliamentary system, the party 
with the most elected members after a federal or 
provincial election normally forms the government. 
If that party has a majority of elected members, it 
can govern without the support of the opposition 
parties. However, if it has only a minority of elected 
members, it usually requires the support of at least 
one other party to pass any legislation, including 
the annual budget. 

Legislative oversight of government spending, 
including the annual budget, is fundamental to 
any democracy. In Canada, such oversight typically 
falls to the opposition parties, although all elected 
officials are ultimately responsible for ensuring 
that revenues are spent prudently on behalf of 
the public. 

The annual budget is generally regarded as a 
government's most important piece of legislation 
each year given that it outlines the government's 
fiscal direction and its policy priorities. It also sets 
out how much the government proposes to spend 
on the services it provides to citizens, and how it 
will fund the cost of providing these services. From 



a fiscal accountability perspective, the budget's 
"bottom line" is the projected surplus or deficit for 
the year. 

The government of Ontario currently spends 
about $120 billion a year on public services ranging 
from health care to education to social assistance. 
While the government sets spending priorities and 
manages service delivery, it must seek the Legisla- 
ture's approval each year for its spending plans. 

In jurisdictions that follow the Westminster-style 
parliamentary system, such as Canada, the United 
Kingdom, Australia and New Zealand, parliament- 
ary oversight and approval of a government's 
proposed spending plans has long been considered 
a key element to ensure government fiscal account- 
ability. Legislative scrutiny of proposed government 
spending, especially by opposition members, is a 
key component of this accountability. 

THE BUDGET, ESTIMATES AND THE 
LEGISLATIVE SPENDING REVIEW 
PROCESS 

In recent years, the Ontario budget has usually been 
presented to the Legislative Assembly in late March 
or early April. However, the budgetary process 
starts several months earlier, when the government 
provides ministries with broad spending guidelines 
that reflect its policy priorities and fiscal outlook. 
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The ministries then put forward their proposed 
spending plans for each of the public services that 
they manage. Ministry plans, once reviewed and 
approved by the Treasury Board and Management 
Board of Cabinet, form the basis for the budget 
and for expense projections of future years. The 
budget and accompanying fiscal plan also reflect 
the expected revenues available to support these 
proposed expenditures. 

Ministry spending plans are subdivided by 
program area and spending activity. Each program 
area, such as post-secondary education or adult 
social services, is allocated a vote number, and 
each spending activity within a particular vote, 
such as grants to colleges and universities or adult 
employment-support assistance, is allocated an 
item number. These proposed votes and items are 
summarized in a document entitled Estimates, and 
represent the government's formal request to the 
Legislature for spending approval. The government 
must table its Estimates no later than 12 sessional 
days after the budget's release. 

In Ontario, a separate legislative committee 
called the Standing Committee on Estimates (Com- 
mittee) is responsible for reviewing the estimates 
of at least six but not more than 12 ministries or 
government offices each year. The Committee is 
currently composed of members from Ontario's 
three major political parties, with representation 
proportionate to the percentage of members each 
party has in the Legislature. During the review pro- 
cess, ministers and senior staff of those ministries 
selected for review appear before the Committee to 
explain their estimates and answer questions. 

When the Committee completes its review, it 
reports back to the Legislature and the estimates 
are then deemed to have been approved by the 
Committee, even if the Committee did not formally 
endorse them. Estimates of those ministries or 
offices not selected for review are deemed to be 
automatically approved. 

Members of the Provincial Parliament (MPPs) 
also have the opportunity to debate the tabled 
Estimates in the Legislature. After the Legislature 



approves them, through passage of the Supply Act, 
they become voted appropriations and constitute 
the legal spending authority for the government. 
The vote-and-item details provide a mechanism 
for legislative control of this proposed spending, as 
spending of the funds must be consistent with the 
stated purpose of each approved vote and item, and 
the amounts spent cannot exceed voted amounts 
without further legislative authority. 

PAST EXAMINATIONS OF ONTARIO'S 
ESTIMATES REVIEW PROCESS 

In 1987, we asked a number of MPPs for their views 
on the estimates review process. We then reported 
in our 1987 Annual Report that the MPPs generally 
felt that the process was "maligned and ineffect- 
ive." We recommended at the time that considera- 
tion be given to creation of a dedicated committee 
mandated to review the annual estimates of 
selected ministries. 

The following year, the Standing Committee 
on Public Accounts issued a Special Report on the 
Estimates Process, and in 1989, the Legislature 
established the current Standing Committee on 
Estimates (Committee). 

After this Committee had operated for about 
five years, we decided in 1995 to revisit this area 
to determine whether there had been progress 
in improving the estimates review process. The 
consensus among the nine MPPs we interviewed — 
three from each party — was that the process was 
still not very effective in ensuring that ministry 
spending plans were appropriately scrutinized. 
In our 1995 Annual Report, we quoted a comment 
from one MPP as indicative of the views we heard: 

We do not serve the public very well in 
assessing whether the ministry is plan- 
ning to spend or has spent the money 
wisely. This disturbs me. In theory, we are 
supposed to be looking after the finances 
of the province. However, by the end of 
the day we are no more enlightened. 



Legislative Estimates Review Process 



Objective and Scope of 
Current Review 



■ 

We decided to revisit the legislative estimates review 
process this year because it has been more than 15 
years since our last review. In addition, our research 
found that other Westminster-style parliaments 
have acknowledged, just as Ontario's has, that there 
have been challenges in implementing effective 
legislative oversight of this area, and a number have 
carried out changes to try to address them. It was 
our view that Ontario's elected members might find 
some of the practices in use elsewhere of interest. 

Aside from our review of practices elsewhere, 
the main focus of our study was to obtain the views 
of MPPs with several years of experience on the 
Standing Committee on Estimates. Accordingly, 
we interviewed nine current members — again, 
three from each party — who had served or are 
currently serving on the Committee. We also took 
the opportunity to obtain the views of three retired 
MPPs from different parties, each of whom served 
a significant number of years as an elected member 
and whose collective experience in the Legislature 
totalled more than 80 years. 

We understand that the Legislature's Standing 
Orders — which govern the workings of the House 
and its Committees, including the Standing Com- 
mittee on Estimates — are currently under review, so 
the observations from our review may prove timely. 



Overall Observations 



The consensus of almost all of the members we 
interviewed echoed the comments members made 
in 1995 when we last examined the estimates 
review process — the process is still not very effect- 
ive in providing meaningful scrutiny of government 
spending plans. However, members also made it 
clear that the hearings of the Standing Committee 



on Estimates are still a worthwhile exercise because 
they provide the only real opportunity to directly 
question a minister outside of question period. 

We asked the members a point-blank question: 
Was the whole exercise little more than a "rubber 
stamp" with respect to overseeing proposed govern- 
ment expenditures? Most acknowledged that, to 
some extent, it probably was. We heard comments 
along the following lines: 

The estimates review process can be a 
very frustrating exercise when you sit 
there for up to 80 hours and not look in 
any detail at the expenditures reported 
in the Estimates. In reality, the process is 
a tool for political purposes — it has noth- 
ing to do with review of expenditures for 
reasonability. 

I do not feel the Estimates Committee is 
meeting the objective of financial scrutiny, 
and it doesn't provide for a better Ontario. 
I give it a scale of two out of 10. 

But on the other hand, members were unani- 
mous in acknowledging the process's value as an 
opportunity for questioning ministers about gov- 
ernment policy and related proposed expenditures. 
As several MPPs put it: 

The estimates review process is a great 
opportunity to get questions that are 
important to you answered and is a good 
learning opportunity. 

You learn a lot from the Estimates. 
Sometimes you will see that members 
will ask questions that the minister is not 
expecting, and it's good to see how they 
respond. 

While it's not perfect, in aggregate it's a 
good thing. 

Our research did not identify any Westminster- 
style parliaments that appeared to have devised a 
truly effective estimates review process, and many 
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are wrestling with the same concerns and issues 
expressed by Ontario MPPs. Indeed, the Inter- 
national Monetary Fund noted in a 2005 study that 
Westminster-style legislatures in general have very 
limited oversight influence, and ranked the parlia- 
ments of Canada, Australia, the United Kingdom 
and New Zealand as having the weakest budgetary 
influence of those in any western countries. The 
legislatures in the United States and Sweden, both 
non- Westminster systems, ranked the highest in 
budget oversight powers. 

A common observation elsewhere was that 
since Committee members cannot realistically 
amend spending proposals, they see little point in 
giving the numbers more than a superficial look, 
regardless of whether they review the estimates in a 
separate estimates committee or in a series of policy 
committees. The research consensus was that 
elected members must first believe their scrutiny 
can influence government spending decisions, pro- 
grams and services before the review process can 
fully engage them. A 2006 report prepared by the 
United Kingdom's Hansard Society, an independ- 
ent, non-partisan educational charity that promotes 
effective parliamentary scrutiny, summed up the 
perceived effectiveness of the United Kingdom's 
process as follows: 

In the view of many commentators, 
Parliament's influence over government 
proposals for taxation and expenditure, 
and priorities within that expenditure, is 
virtually non-existent. The essential rela- 
tionship between Parliament and govern- 
ment is that the latter proposes and the 
former simply agrees. To draw an analogy, 
the government decides the value of the 
cheque, to whom it should be paid and 
when, and Parliament simply signs it. 

Ontario MPPs raised the following specific issues 
or concerns, which our research suggested were 
also common elsewhere: 

• Elected members have numerous and compet- 
ing demands on their time, especially when 



the Legislature is sitting, and so would not 
welcome any changes requiring them to add 
to the already significant time they spend on 
estimates. However, most MPPs we spoke with 
said it should be possible to identify changes 
that would make the time they currently 
spend on estimates more effective. Specific- 
ally, members said: 

We do not get good value for all 
the time spent in Estimates. Surely, 
there are some changes that could 
be made which, while not adding 
more committee time, would result 
in a better process. 

Whatever you recommend, do 
not include anything that adds 
more time to the estimates review 
process because we just do not have 
the time. 

• Ministry briefing materials to support the 
estimates review process are quite detailed but 
are still perceived as not providing, in an easily 
understandable manner, adequate information 
on how a program is being delivered and what 
results are being achieved. This makes it dif- 
ficult for Committee members to evaluate ser- 
vice delivery relative to funds being requested. 
The following two comments reflect some of 
the feedback we heard on this issue: 

Unless someone knows the ministry 
in depth, you really cannot make 
heads or tails of the Estimates. 

Members should read the Estimates 
in detail, but no one really reads 
them. Estimates are complicated to 
read. It is very difficult for a non- 
financial person to read and sort 
through them. 

• Often, a significant amount of the requested 
funding may have already been spent before 
the Estimates are reviewed. MPPs said this 
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raises questions about whether the purpose 
of the exercise is to have members scrutinize 
the government's proposed spending plans 
or, instead, to give MPPs access to ministers. 
Members observed that: 

The chamber is here to represent 
the views of Ontarians. Therefore, 
the estimates review process is 
needed, but how it should work 
is the question. There should be 
some ability for the Committee to 
identify problems and report on 
recommendations, but this does not 
happen. So the estimates review 
essentially becomes what I want to 
use it for — taking up the concerns 
of my constituents. Essentially, the 
Estimates becomes a place to get 
information on those constituency 
issues you need to address. 

The roles of the Committee should 
be to review, quantify and qualify 
government expenditures and to be 
in a position to make amendments 
to the government's spending plan, 
where appropriate. The Committee, 
along with the Legislature, should 
hold the government to account on 
the spending of tax dollars. The pro- 
cess should be about transparency. 
But in reality the process is a bal- 
ance between politics and numbers. 
Therefore, the Committee needs 
an upgrade — a review of what it is 
doing and why it is doing it. 

In answer to our question about partisan- 
ship, the MPPs said that the estimates review 
process would likely be more effective if the 
partisanship could be dialed down. However, 
all MPPs recognized that this is easier said 
than done since legislative committees are 
composed of both opposition and government 
members. As two MPPs commented: 



There are a lot of talented people, 
but the politics remove the ability 
of the Committee to undertake any 
meaningful scrutiny. 

The adversarial dynamics in any 
legislative committee comprised of 
both opposition and government 
members will almost always foster 
an atmosphere of partisanship. As 
a result, government members ask 
questions that are intended to make 
the government look good, while 
opposition members ask questions 
intended to do the opposite. 

In general, the three long-serving retired MPPs 
we spoke with echoed the sentiments of the nine 
current members interviewed that the legislative 
estimates review process was not seen as being that 
effective as a means of scrutinizing government 
spending. While they also acknowledged the bene- 
fit of having the opportunity to directly question 
a minister, one retired MPP made the following 
observation: 

I like Public Accounts because the Min- 
ister is not there. One change I would 
make: take the Minister out of the pro- 
cess, as this is the only way committee 
members will get legitimate and straight- 
forward answers. 

The retired MPPs, who have witnessed the 
evolution of the estimates review process over 
time and have been both committee members and 
ministers, felt that the process has become less 
effective as the size of government has grown. 
They also noted that any reform had to go beyond 
administrative changes. Specifically, the govern- 
ment needed to be more open in responding to 
questions, and committee members needed to 
focus both on financial scrutiny of the Estimates as 
well as on policy-related issues. One of the retired 
MPPs expressed the following opinion: 
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The concentration of power without 
a counterbalance is the problem with 
government today — that is why you need 
the Standing Committee [on Estimates] . 
If you want to get accountability, it is 
important for the Committee to decide, 
where do we want to go with this issue 
and how do we get there. 



Detailed Observatio 



i 



THE WESTMINSTER SYSTEM OF 
GOVERNMENT 

The ability of a legislature to influence a govern- 
ment's budget depends on the balance of power 
between the legislative and executive branches. 
Ontario follows the Westminster-style system, 
which emphasizes the executive's right to govern. 
Budgetary power is accordingly concentrated 
within the government, and the parliament as a 
whole has only minimal influence over it, especially 
in majority situations. Noting this, the International 
Monetary Fund (IMF) ranked the legislatures of 
Canada, Australia, the United Kingdom and New 
Zealand in a 2005 study as having the weakest 
budgetary oversight powers among major western 
countries, as shown in Figure 1. 

The federal Gomery Royal Commission, which 
in 2004 launched an investigation into the so- 
called Sponsorship Scandal in the federal govern- 
ment, also touched on the issue of Parliament's 
ability to effectively oversee a government's spend- 
ing plans, observing: 

Concerns about the effectiveness of 
Parliament in examining government 
programs and discharging its financial 
accountability role have been expressed 
for decades. Most recently, a 2003 com- 
mittee report [Report of the Government 
Operations and Estimates Committee 



Figure 1: International Monetary Fund Index of 
Legislative Budgetary Powers* 

Source of data: IMF (2005) 



United States 
Sweden 
Italy 
Japan 
Austria 
Finland 
Netherlands 
Norway 
France 
Germany 
Spain 
Australia 
Canada 
United Kingdom 
New Zealand 







10 



* This figure measures tile relative power of legislatures over national budgets 
on a scale of one to 10, with 10 representing significant power. The IMF 
considered whether legislatures: (a) approve an annual budget strategy; 
(b) have powers to amend draft budgets; (c) allot time for discussion of the 
annual budget; and (d) receive technical support for scrutinizing the budget. 

(Canada) entitled Meaningful Scrutiny: 
Practical Improvements to the Estimates Pro- 
cess] lamented that "committees continue 
to provide relatively cursory attention to 
the main spending estimates and explana- 
tory reports," and MPs interviewed for 
a Commission research study confirmed 
this finding. 

In contrast to Westminster-style democracies, 
the IMF ranked the legislatures of the United States 
and Sweden as having the strongest budgetary 
oversight powers. However, this power balance 
also has its disadvantages. In the United States, for 
example, budget debates can lead to paralyzing 
political gridlock, as when Congress threatened in 
2011 to defeat the Administration-backed budget 
bill and effectively shut down the government 
unless certain of its demands were met. 



DEDICATED ESTIMATES COMMITTEE 
VERSUS POLICY-FIELD COMMITTEES 

Ontario is the only Canadian jurisdiction that 
has established a dedicated Standing Committee 
on Estimates, as illustrated in Figure 2. All other 



Legislative Estimates Review Process 



Figure 2: Provincial Comparison-Estimates Review Time and Committee Type 

Prepared by the Office of trie Auditor General of Ontario 





Time Given for Estimates Review 


Committee Type 


British Columbia 


125 to 200 hours (on average) 


Committee of the Whole 1 


Alberta 


53 hours (minimum) 


Policy-field committees 2 


Saskatchewan 


75 hours (minimum) 


Policy-field committees 


Manitoba 


100 hours (maximum) 


Committee of the Whole 1 


Ontario 


90 hours (maximum) 


Standing Committee on Estimates 


Quebec 


200 hours (maximum) 


Policy-field committees 


New Brunswick 


80 hours (maximum) 


Committee of the Whole 13 


Nova Scotia 


80 hours (maximum) 


Committee of the Whole 14 


Newfoundland and Labrador 


75 hours (maximum) 


Policy-field committees 


Prince Edward Island 


None specified 


Committee of the Whole 1, 5 


Yukon 


None specified 


Committee of the Whole 1, 5 


Nunavut 


None specified 


Standing Committee on Oversight of Government 
Operations and Public Accounts 5 



1. This term is used to describe the Legislative Assembly, which, in essence, is a committee of all elected members. 

2. Three hours for each department and two hours for the Executive Council. 

3. The Legislative Assembly may refer Estimates to a dedicated Standing Committee on Estimates, but this occurs only rarely. 

4. Forty hours for each of the Committee of the Whole on Supply and the Subcommittee on Supply. 

5. Information for Prince Edward Island, Yukon and Nunavut was not readily available. 



provinces have assigned this oversight role to their 
policy-field legislative committees or to a commit- 
tee of the whole Legislature. Typical policy fields 
often comprise those ministries that share a com- 
mon theme such as justice, and which cover, for 
example, such areas as police, correctional facili- 
ties, probation and parole. 

One benefit of having a distinct estimates com- 
mittee is that parliamentarians get dedicated time to 
review and debate the proposed spending of at least 
some ministries. On the other hand, the policy-field 
committee approach often allows members to gain 
more detailed knowledge about certain ministries 
and programs, making them better informed about 
both the issues and related program-expenditure 
trends. However, a drawback of the policy-field 
approach is that these committees are often also 
required to review draft legislation or perform other 
tasks. Only part of their time maybe spent actually 
reviewing ministry estimates. 

The estimates review in the major Westminster- 
style countries also tends to be done by policy-field 
committees rather than dedicated estimates com- 



mittees, as illustrated in Figure 3. We noted from 
our research that, irrespective of the approach 
taken to review estimates, all jurisdictions indicated 
similar challenges in legislative scrutiny of their 
government's proposed spending plans. 

Ontario's 90-hour maximum time allocation for 
estimates review is fairly consistent with that of the 
other provinces, as illustrated in Figure 2. How- 
ever, our research found that, like Ontario, other 
Canadian jurisdictions do not use all of the allotted 
time. In Ontario, the Estimates Committee typically 
spends about 50 hours in hearings. MPPs told us 
there were several reasons why the full 90-hour 
allocation was not used: 

• There are competing demands on members' 
time that include attending debates in the 
Legislature, conducting other committee 
work and addressing party-caucus and con- 
stituency issues. 

• In recent years, the Estimates have not been 
tabled until mid-April, leaving limited time for 
committee meetings before the summer recess 
in mid-to-late June. 
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Figure 3: National Comparison-Estimates Review Time and Committee Type 

Prepared by the Office of the Auditor General of Ontario 





Time Given for Estimates Review 


Committee Type 1 


Canada 


Three months 


Policy-field committees 


United Kingdom 


14 days (recently increased from eight days) 


Policy-field committees 


Australia 


Four days per committee 


Policy-field committees 


New Zealand 


Two months 


Policy-field committees 



• The Estimates Committee normally meets 
only when the Legislature is sitting and 
when its day's routine proceedings have been 
completed. 
Given that devoting more time to estimates 
review is not an option for most committee mem- 
bers, any improvements must focus on making the 
best use of what time is available. 

QUALITY OF BRIEFING INFORMATION 

Ministries selected to appear before the Estimates 
Committee provide MPPs with supplementary 
material in the form of briefing books. Since many 
MPPs lack an accounting or financial background, it 
is essential that this material be easy to understand, 
identify major program changes and spending 
trends and, perhaps most importantly, be reason- 
ably concise. Several MPPs commented that it takes 
several years sitting on the Estimates Committee 
before they are able to fully grasp the ministries' 
briefing materials. As one of the retired MPPs with 
whom we met stated: 

The way the material is presented is so 
important because it frames the issues. 
If presented well, it will allow both the 
experienced member and the average 
member to gain the necessary knowledge 
to understand the program and then to 
ask the right questions. 

Legislature rules provide for some consistency 
in the preparation of briefing books, requiring 
that they include "information on growth rates, 
interim expenditures for the previous fiscal year, 



and an explanation of the programs and funding 
by a particular item." The Treasury Board provides 
guidelines to help ministries comply with these 
requirements. Despite this, however, we found 
that the information in briefing books often did 
not clearly identify or discuss expenditure trends 
or address significant changes to programs beyond 
one year. As well, briefing books tend to focus on 
government initiatives but provide little quantifi- 
able information on program results. 

Interviews with MPPs led us to conclude that 
most MPPs did not spend much time reviewing the 
briefing books in advance of committee meetings 
because they did not find the information all that 
useful. Several MPPs advised us that they would like 
more performance and future-oriented information 
for programs with significant changes to spending 
or service delivery so as to better assess the impact 
of the proposed changes. We believe this suggestion 
has considerable merit. In particular, MPPs told us 
that information on program reductions is particu- 
larly relevant in this era of cutbacks and austerity. 

As a recent Ontario ministry publication dealing 
with performance measurement at the municipal 
level put it: 

Government today is very complex, so it is 
important that elected officials and public 
servants inform taxpayers what govern- 
ments plan to achieve, what it is actually 
accomplishing and what public services 
cost. With this information, taxpayers 
can make informed decisions about the 
level of services they desire. This notion of 
accountability is fundamental to our form 
of government. 
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Program Performance Information 

Clearly, MPPs need accurate, transparent and 
understandable information to help them assess 
whether government spending is meeting its stated 
objectives. In this regard, we believe Ontario's 
Standing Committee on Estimates would find brief- 
ing materials more useful if they contained: 

• measurable targets for program objectives; 
and 

• a comparison of actual performance with 
forecasts. 

Many other jurisdictions have recognized this 
need since our last review in 1995. We reviewed 
a number of publicly available annual plans and 
performance reports published by departments and 
ministries in Alberta, British Columbia, Canada, 
Australia and the United Kingdom. Most jurisdic- 
tions reviewed provided more comprehensive plan- 
ning reports that laid out strategy, activities and 
performance measures with stated targets. Subse- 
quent annual performance reports explicitly com- 
pared actual performance with prior targets and 
discussed variances. The majority of jurisdictions 
reviewed also provided plans, expense projections 
and performance targets for three to four outlook 
years, as illustrated in Figure 4. 

It is notable that the United Kingdom has 
recently reformed its performance reporting prac- 
tices. As part of major reforms to the public service 
following the 2010 election, the government abol- 
ished the previous reporting regimen of centrally 
defined Public Service Agreements. In its place, 
a five-year strategy was developed, and depart- 
ments annually develop business plans that set out 
proposed actions to implement the government's 
strategy. Subsequent performance reporting tracks 
these actions. Interestingly, targets are no longer 
formally set. Departments also report on several 
indicators that are meant to help monitor perform- 
ance. The development of these indicators was still 
ongoing at the time of our review. 

As the chair of the U.K. Public Accounts Commit- 
tee recently said: 



It is not good enough to dump data into 
the public domain. It must be analysed to 
be relevant, robust and fit for purpose. 

The MPPs we interviewed who commented 
on this issue generally were of the opinion that 
they required "more relevant" rather than "more" 
information. 

Future-oriented Information 

Although the Ontario budget provides projected 
future expense information, the Estimates Com- 
mittee has a mandate to review only current-year 
spending plans. Consequently, ministry briefing 
books do not provide information or projections 
regarding the next few years. 

A 2006 study by the Parliamentary Informa- 
tion and Research Service of Ottawa's Library of 
Parliament suggested that federal parliamentary 
policy-field committees reviewing estimates should 
expand their focus to include future spending 
plans. They argue that the government likely 
would be more receptive to recommendations that 
touched on future spending because criticism of 
the more detailed current-year plans could create a 
non-confidence issue. We believe this observation 
has some pragmatic merit; committee members 
would likely be more motivated to focus on spend- 
ing and the level of services actually being provided 
if ministries and the government considered their 
input more seriously. 

At the time of our 1995 review, only a few juris- 
dictions were providing future-oriented estimates 
information. Today, however, much more of this 
information is available, as illustrated in Figure 4. 
The results of these initiatives have been mixed: 
• On the positive side, the U.K. Hansard Society 
noted in 2006 that scrutiny of departmental 
annual reports has enabled committees to 
conduct more comprehensive analyses of gov- 
ernment spending priorities both within and 
across departments. 
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Figure 4: Estimates Briefing Information: Other Jurisdictions 

Prepared by the Office of the Auditor General of Ontario 



Jurisdiction Program Performance and Future-oriented Information 


British Columbia 


• The government releases an annual strategic plan of priorities, objectives and expected results 
over the next three years. 

• Annual departmental service plans provide three-year business plans consistent with the 
government's strategic plan and outline specific performance measures with targets for all 
outlook years. 

• Annual departmental service plan reports compare prior year's actual performance against 
targets set in the previous service plan. 


Alberta 




• Ministries provide a three-year business plan with goals, priority initiatives and performance 
measures with targets for all outlook years. 

• Annual ministry reports track key performance measures and discuss the prior year's 
performance against targets set in the prior year's business plan. 


Canada 




• An annual ministerial Report on Plans and Priorities outlines forecast spending and expected 
results over three years, with stated performance indicators and overall targets. 

• Departmental performance reports compare the prior year's actual performance against plan. 

• In the fall, the targets in the ministerial Report on Plans and Priorities are compared to actual 
departmental reports, which allow for feedback and adjustments before the next Estimates are 
tabled in the spring. 


Australia - 


Queensland 


• Departmental Strategic Plans outline a department's agenda over the next four years. 

• Departmental annual reports provide an account of the operational and financial performance 
as well as an assessment of performance against the Strategic Plan. 


United Kin 


gdom 


• In November 2010, the coalition government directed all departments to develop five-year 
business plans that detail the department's strategy for implementing the government's core 
priorities. 

• Departmental business plans are updated annually and describe plans and actions; in 2012, 
expenditures were forecast out to 2015. 

• Departmental annual reports detail the progress of actions outlined in the prior year's business 
plan and report on input and impact indicators that are used to monitor performance. 



Several Canadian and U.K. studies have noted 
that the provision of future-oriented informa- 
tion has resulted in a more effective scrutiny 
process. But other studies also found that 
estimates documentation, including supple- 
mental planning and performance reports, 
continues to be of poor quality. 
In 2002, the Alberta Financial Management 
Commission, an expert panel set up by the 
province to examine its financial and budget- 
ing practices, noted that ministries tend to 
treat future years merely as mathematical 
exercises. 



While some MPPs we interviewed supported 
the inclusion of future spending plans in estimates 
briefing materials, others opposed it. One argument 
against including future fiscal outlook data is that 
such scrutiny may not be meaningful since govern- 
ment plans may change significantly in subsequent 
budgets. On the other hand, as one member noted, 
suggestions on future spending plans might find 
some resonance with a minister or senior ministry 
officials even though any public acknowledgement 
of this is unlikely to be forthcoming. As one of the 
retired MPPs we spoke to put it: 
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If you are participating in estimates 
review as a Minister, it may not change 
things, but it is very useful for oversight 
because as Minister I may begin to rethink 
some of the program decisions based on 
the questions I am getting. 



NARROWING THE FOCUS FOR PART OF 
THE ESTIMATES REVIEW 

Another way to enhance legislative spending 
control is to narrow the Estimates Committee's 
oversight focus by selecting only a few ministry pro- 
grams for more detailed scrutiny. This idea was put 
forward by the Library of Parliament study referred 
to above. The premise behind this idea is that the 
Committee just does not have the time to conduct a 
detailed scrutiny of all programs administered by a 
large and complex government ministry. 

However, adopting such a targeted strategy is 
not without risk, as important issues might well 
go unaddressed. It would also mean that the Com- 
mittee's ultimate approval or rejection of ministry 
estimates as a whole would be based on a review 
of only a portion of its activities. Despite these 
concerns, several of the MPPs we interviewed said 
this approach might have some merit for part of the 
time allocated to a particular ministry's estimates. 

In essence, this method would employ a two- 
track approach. The first part of the review would 
be a general session with the minister in which 
committee members could ask questions about 
any ministry programs and activities, while the 
second would focus on specific programs. Selecting 
programs for review in more detail could be done in 
one of two ways: 

• a subcommittee composed of one member 
from each party would choose from one to 
three programs; or 

• each of the three parties would select a pro- 
gram on which to focus. 

These choices would be made in advance so that 
the ministry briefing books could provide more 
detailed information about the programs selected. 



This information could usefully include 10-year his- 
torical and future-year expenditure trends, data on 
service levels being provided, outcomes achieved 
and the fiscal impact of any recent policy changes 
and future expenditure trends on the expected level 
of service demand. 

Several MPPs told us that it might not be 
necessary for the minister to be present during 
this portion of the estimates review. Some cited 
as an example the Standing Committee on Public 
Accounts, where only senior ministry and broader- 
public-sector officials are questioned. Those 
familiar with this committee said such an approach 
seemed to work well and tended to de-emphasize 
the partisan nature of committee hearings. This 
approach would focus the second track of estimates 
hearings on the fiscal and administrative delivery 
aspects of the program rather than on the merits of 
government policy, a matter best left for the minis- 
ter's appearance. 

Two MPPs who felt this two-track approach 
might warrant consideration said: 

I believe each of the parties should be 
asked what to focus on. This way, more in- 
depth analysis on policy and expenditures 
will occur rather than have the estimates 
review be wide open. This would likely 
lead to a better understanding of govern- 
ment spending than is being obtained 
from the current estimates review process. 

I'd like to see it. It is better to zero-in on 
specific areas of concern .... 

One MPP who supported the two-track option 
did offer a caution, however: 

I am not in favour of limiting what can be 
reviewed with the minister or ministry 
staff during the estimates review process. 
What is currently open to scrutiny should 
be left open. 

One of the retired MPPs noted: 

There is no way the committee can 
effectively scrutinize the Estimates today 
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because the programs are just too big 
and complex for committee members to 
understand and know what questions to 
ask. Having served as both a member of 
the committee and as a former Minister 
who was on the receiving end of ques- 
tions, I am a big believer in narrowing the 
focus of the review. 

ADVANCE BRIEFINGS 

Another option we raised with MPPs was the value 
of a 30-minute to one-hour in camera briefing 
prior to the public committee hearing to provide an 
overview of the selected ministry's estimates. For 
instance, this briefing could cover any major chan- 
ges in program expenditures or service delivery 
levels, per capita cost comparisons with other prov- 
inces that had similar programs, or other issues 
such as the service and fiscal impact of recent 
policy changes. We suggested the briefing could 
be done either by the Legislature's research staff 
or by senior ministry officials — or possibly even 
by our Office, if we had done a recent audit in that 
program area. The Standing Committee on Public 
Accounts has effectively utilized such private in 
camera briefings in advance of the public hearing 
for most of the past decade. 

The 2003 report of the federal Government 
Operations and Estimates Committee, entitled 
Meaningful Scrutiny: Practical Improvements to the 
Estimates Process, made a recommendation along 
these lines. It suggested that the committee could 
leverage the work of the Auditor General's office by 
obtaining its advice when reviewing the estimates 
of recently audited departments and agencies. 

The majority of MPPs we spoke to felt an advance 
briefing would be useful. Among their comments: 

It would be very helpful to have a one- 
hour briefing from a non-partisan official 
or even a ministry official that might 
explain why spending is up by 6% in one 
area and down 4% in another. Even a 
briefing from the Auditor General indicat- 



ing, "Here is what I noted ..." would go 
a long way in helping members do their 
job ... by pointing out trends, it would 
trigger questions we should be asking 
about what is happening and, more 
importantly, why is this happening. 

If someone can brief us in a non-partisan 
way, then yes this would be a great idea . . . 
We need non-partisan and factual brief- 
ings. Perhaps the deputy minister or assist- 
ant deputy minister can give the briefing. 

Yes. This would be great. Currently, too 
much time and emphasis is spent on the 
minister's response. If the Auditor General 
gave a briefing, maybe this would not be 
in the best interest for him, but it would 
be better for the committee, given his 
independence. 

SPECIALIZED SUPPORT 

Ministries spend billions of dollars across dozens 
of individual programs and activities each year. 
However, while the Legislature's Public Accounts 
Committee, for example, has a dedicated legislative 
research officer available to assist committee mem- 
bers, the Estimates Committee does not. Without 
such support, and considering the time demands 
on MPPs, it is more difficult for members to fully 
understand a ministry's estimates or obtain answers 
to fundamental questions such as: "Is public money 
being spent effectively? Are desired outcomes being 
achieved? Should government spending priorities 
be revised in light of new events or circumstances?" 

It is equally difficult to identify those areas 
where proposed spending may not be appropriate 
relative to the level of service being delivered, or 
where significant variances in spending over time 
warrant additional scrutiny. 

Our research indicated that most legislatures in 
Westminster-style parliamentary democracies have 
minimal, if any, dedicated research support. By 
contrast, the U.S. Congressional Budget Office has 
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a large research arm of accountants and economists 
available to help legislators analyze and assess the 
government's proposed budget. 

The MPPs we spoke to generally agree that 
having research assistance would help them better 
understand the estimates briefing materials. In 
fact, the federal governments of Canada, the United 
Kingdom and Northern Ireland, Scotland and Wales 
have moved to address this need in recent years, as 
illustrated in Figure 5. 

In its 2006 report, entitled The Fiscal Maze, the 
U.K. Hansard Society pointed to the success of the 
United Kingdom's House of Commons Scrutiny 
Unit, established to provide specialized support for 
select parliamentary committees, as evidence that 
offering resources and research can contribute to 
more effective financial review. The society advo- 
cated expanding the unit's role to provide compre- 
hensive support for all committees. 



Three Things To Consider 



We offer the following ideas for consideration. 
1. A short in camera estimates briefing for 
members in advance of formal committee 
sessions for each selected ministry. 



The briefing would cover such areas as any 
significant changes in program expenditures 
where relevant, expenditure trends over time 
and expenditures relative to other provinces, 
given that most provinces deliver similar ser- 
vices to their citizens. It could be conducted 
by legislative research staff, by senior ministry 
officials or perhaps even by our Office. 

2. Requesting that ministry briefing books 
combine information on spending plans 
with past and current performance 
reporting to provide a better basis for 
legislators to assess what results are being 
achieved for proposed expenditures. 

Specifically, the briefing materials would 
provide historical trend information over a 
five- to 10-year period and include program 
and service delivery indicators as well as 
future-oriented outlook data, at least for the 
most significant ministry programs. How- 
ever, this should not result in any significant 
increase in page count, and the readability 
and understandability of briefing materials 
should be a priority. 

3. Selection of a few specific ministry pro- 
grams for more in-depth review by the 
committee, perhaps with only senior min- 
istry officials in attendance. 



Figure 5: Specialized Support: Other Jurisdictions 

Prepared by the Office of trie Auditor General of Ontario 



Jurisdiction Specialized Support Available 



Canada The Library of Parliament's Parliamentary Information and Research Service provides research support. 

In the past, it has hired analysts to provide additional support to committees doing estimates work. 
Such support includes: 

• briefings on the Estimates and the estimates process; 

• background research on programs and activities; and 

• analysis of plans and briefing notes to help Committee members prepare for meetings and pose 
appropriate questions. 

The Parliamentary Budget Office also produces additional reports that can be used by committees. 

United Kingdom Recent reforms include the formation of a Scrutiny Unit that has a staff of 15, including economists, 
lawyers and accountants. The unit provides select committees with financial and legislative expertise, 
identifies areas of interest and suggests lines of inquiry. 

Northern Ireland, Specialist researchers within the research unit of the governing body are available to provide guidance 
Wales and Scotland on financial and technical matters. 
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Given that the larger ministries spend bil- 
lions of dollars in dozens of different areas, a 
more focused review of several key programs 
for at least part of those ministries' time allo- 
cations might have merit. Either a subcommit- 
tee composed of one member from each party 
would select the ministry programs for more 
detailed examination, or each party would 
select one program for review. The minister 
would continue to attend a significant portion 
of estimates hearings so that policy issues in 
any area could still be raised. 
In regard to our proposals, the former MPPs we 
met with responded positively to having advance 
briefings. One suggested that to reduce partisan- 
ship, only ministry staff should be present during 
meetings, to focus questions away from policy and 
toward program administration. However, another 
former MPP disagreed, believing that attendance by 
the minister would be useful. 



When asked about the estimates briefing materi- 
als provided to members, the former MPPs agreed 
that more relevant and future-oriented perform- 
ance information was needed. One said: 

In my view, ministries spend a tremen- 
dous amount of time preparing material 
for briefing books that members do not 
really use — but if properly oriented, esti- 
mates review would have more meaning 
if members had good information on pro- 
gram performance and if the information 
also focused on looking forward rather 
than just looking at the past. 

On the issue of focusing on only a few specific 
programs for part of the review, two of the retired 
MPPs supported this proposal, while one member 
was concerned this could limit the scope of mem- 
ber questions. 



Chapter 3 
Section 




Ministry of Health and Long-Term Care 



Long-term-care Home 
Placement Process 



Background 




Long-term-care homes (LTC homes) provide care, 
services and accommodation to people who require 
the availability of 24-hour nursing care, super- 
vision in a secure setting, or frequent assistance 
with activities of daily living such as dressing and 
bathing. LTC homes are sometimes referred to as 
nursing homes or homes for the aged. They may be 
for-profit, not-for-profit, or municipally run organ- 
izations, and often have waiting lists for their beds. 

In July 2010, the Long-Term Care Homes Act, 
2007 (Act) came into effect upon finalization of 
related regulations. This Act replaced the Nursing 
Homes Act, the Charitable Institutions Act and the 
Homes for Aged and Rest Homes Act. It governs 
the process for placing people in LTC homes, and 
authorizes the province's 14 Community Care 
Access Centres (CCACs) to manage this process 
(see Figure 1 for CCAC boundaries) . Accordingly, 
the CCACs determine eligibility for admission, pri- 
oritize eligible individuals on LTC homes' wait lists 
and arrange placement when a bed becomes avail- 
able. In the 2011/12 fiscal year, the CCACs placed 
more than 25,000 people in Ontario's 640 LTC 
homes, which have a total of 76,000 long-term-care 
beds that are over 97% occupied. About 85% of the 
people placed were aged 75 and older. 



Each CCAC reports to one of 14 Local Health 
Integration Networks (LHINs). In the 2011/12 fiscal 
year, the Ministry of Health and Long-Term Care 
(Ministry) provided $2.1 billion of funding to the 
CCACs through the LHINs. This funding covered 
the CCACs' LTC home placement services, as well 
as their other activities, including the provision of 
home care and community support services. Infor- 
mation was not available on the cost of LTC home 
placement alone. The Ministry, to which the LHINs 
are accountable, is responsible for ensuring that 
CCACs comply with provisions for LTC home place- 
ment under the Act. 



Audit Objective and Scope 



The objective of our audit was to assess whether 
the processes in place at selected CCACs were 
effective for placing individuals in LTC homes 
in a consistent and timely manner, based on 
their needs and in accordance with ministry and 
legislative requirements. 

We conducted our audit work at three Commun- 
ity Care Access Centres of different sizes: Central 
East CCAC (responsible for 9,700 LTC home beds, 
with head office in Whitby); North East CCAC 
(responsible for 5,000 LTC home beds, with head 



185 



2012 Annual Report of the Office of the Auditor General of Ontario 



Figure 1: Community Care Access Centre Boundaries 

Source of data: Ontario Association of Community Care Access Centres 
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3. Waterloo Wellington 
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5. Central West 

6. Mississauga Halton 
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8. Central 

9. Central East 

10. South East 
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12. North Simcoe Muskoka 

13. North East 

14. Northwest 



office in Sudbury) ; and Waterloo Wellington CCAC 
(responsible for 3,800 LTC home beds, with head 
office in Kitchener). Senior Ministry and CCAC 
management reviewed and generally agreed to our 
audit objective and associated audit criteria. 

The scope of our audit included a review and 
analysis of relevant files and administrative policies 
and procedures, as well as interviews with appro- 
priate CCAC and ministry staff. We also reviewed 
relevant research, including best practices for the 
LTC home placement process in other jurisdic- 
tions. In addition, we obtained the perspective 
of the Ontario Association of Community Care 
Access Centres, which represents the 14 CCACs; 
the Ontario Long-term Care Association and the 
Ontario Association of Non-profit Homes and Ser- 
vices for Seniors, which between them represent 
the majority of LTC homes in the province; and the 
Advocacy Centre for the Elderly, which represents 
low-income seniors. We also used computerized 
data extraction techniques to analyze data from the 
Ministry's Client Profile Database, which includes 
LTC home placement information received from 
the CCACs. 




We did not rely on reports from the Ministry's 
internal audit service team because it had not 
conducted any work on the CCACs' LTC home 
placement processes. 



Summary 



Since 2005 the number of Ontarians aged 75 and 
over has increased by more than 20%, which has 
undoubtedly been a key reason why the median 
time that people wait for accommodation in an 
LTC home has almost tripled — from 36 days in 
the 2004/05 fiscal year to 98 days in the 2011/12 
fiscal year. Although wait times have decreased 
somewhat since July 2010, when tighter eligibility 
criteria in the Long-Term Care Homes Act took effect, 
Ontario's population of people aged 75 and up is 
expected to grow by almost 30% between 2012 
and 2021, creating additional pressures to meet 
the needs of people who require long-term care. 
As well, beginning in 2021, the first of the baby 
boomer generation — those born between 1946 
and 1964 — will start to turn 75, at which point the 
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demand for long-term care is expected to become 
even greater. 

While CCACs are responsible for the process 
of placing individuals in LTC homes, numerous 
factors outside their control affect wait times for 
placement. In particular, the Ministry of Health and 
Long-Term Care (Ministry) is responsible for the 
number of available LTC home beds; individuals 
are allowed to select the LTC home(s) that they are 
willing to be placed in; and LTC homes may reject 
applications if they believe their home lacks the 
nursing expertise or physical facilities needed to 
meet the applicant's care requirements. 

Numerous studies have shown that remaining 
in hospital longer than medically necessary is detri- 
mental to a patient's health, yet many people wait 
in hospital for an LTC home bed to become avail- 
able, which occupies a hospital bed that is often 
needed by other patients who have more complex 
health-care needs. As well, occupying a hospital bed 
is more expensive than community-based alterna- 
tives. This situation is exacerbated because people 
can wait in hospital for the LTC home(s) of their 
choice, even if the chosen home(s) have a lengthy 
wait list. We noted that during the 2011/12 fiscal 
year, 19% of clients waiting in hospital had applied 
to only one LTC home. Our research indicated that, 
to minimize the time such patients spend waiting in 
hospital, other provinces have stricter policies: five 
provinces require patients to go to the first vacant 
bed in any LTC home; and two require patients to 
go to any home with an available bed within 60 and 
100 kilometres, respectively. 

Given our aging population, developing alterna- 
tives to long-term care and implementing more 
efficient processes for placing people in an LTC 
home in a consistent and timely manner is critical. 
The Ministry has recognized this and has supported 
a number of initiatives to help reduce or delay the 
need for long-term care, and improve the place- 
ment process when a bed in an LTC home is needed. 
For example, all CCACs use a provincially standard- 
ized process to determine client eligibility, includ- 
ing considering alternatives to long-term care. This 



process also helps determine each client's wait-list 
priority; however, more needs to be done to ensure 
that crisis cases are prioritized consistently. 

All three of the CCACs that we visited were 
managing various areas of their LTC home place- 
ment process well. However, all also had areas 
where improvements could be made, although 
any changes made in these areas would likely not 
significantly improve LTC home wait times. Some of 
our more significant observations are as follows: 

• The provincial agency Health Quality Ontario 
indicated that nearly 20% of the CCACs' 
home-care clients who were subsequently 
placed in LTC homes could have remained in 
the community, and a Ministry-commissioned 
study noted that 37% of clients waiting in hos- 
pital for an LTC home bed had care needs that 
were no more urgent or complex than those of 
people being cared for at home. 

• Not all people eligible for an LTC home 
require such care; for example, all veterans 
and spouses of current residents are eligible 
regardless of their health-care needs. 

• In the 2011/12 fiscal year, CCACs province- 
wide completed a total of about 36,000 formal 
client reassessments, which are required to be 
completed in six-month intervals and within 
three months of LTC home placement. How- 
ever, conducting a quick "touch-base" with 
clients and their families might more quickly 
and cost effectively provide information on 
whether a client's condition has changed 
enough to warrant a formal reassessment 
rather than conducting reassessments every 
six months. 

• March 2012 LTC home wait-list data indicated 
crisis clients had waited a median of 94 days 
up to that point; moderate-needs clients 

had waited 10-14 months; and most other 
eligible clients had been on the wait list for 
years. Further, during the 2011/12 fiscal 
year, 15% of clients died before receiving LTC 
home accommodation. 
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• While 36% of clients were placed in their first 
choice of homes, others generally accepted 
the offered home but remained on the wait 
list for their preferred home(s). In fact, in 
March 2012, 40% of people on the wait list 
already resided in long-term care. At least 
half of admissions to more than 70 LTC homes 
during the 2011/12 fiscal year were for crisis 
clients, who typically get priority for the 
home of their choice. Consequently, non-crisis 
clients may find it difficult to access accommo- 
dation in the newer or more popular homes. 

• The CCACs visited did not periodically review 
client placement decisions to ensure that the 
highest-priority person meeting an available 
bed's criteria was offered the bed. Nor did the 
CCAC systems retain wait-list information so 
that these decisions could be reviewed after 
the fact. 

• Applicants living in some areas of the prov- 
ince get into LTC homes more quickly. At one 
CCAC, 90% of clients were placed within a 
low of 317 days, whereas at another CCAC, 

it took about 1,100 days until 90% of clients 
were placed. 

• While LTC homes can designate up to 60% 
of their beds as preferred accommodation 
(that is, private or semi-private), only 40% 
of clients apply for these more costly beds. 
Therefore, regardless of care needs, clients 
who can afford to pay for preferred accommo- 
dation tend to get placed more quickly than 
other clients. 

• Although information on LTC home wait 
times by priority level or accommodation 
type (that is, private, semi-private and basic) 
would help people consider where to apply, 
only one CCAC we visited made some of this 
information available publicly. 



| OVERALL MINISTRY RESPONS 




The Ministry of Health and Long-Term Care 
(Ministry) welcomes the advice contained in 
this value-for-money audit. The audit recognizes 
a 20% increase since 2005 of Ontarians aged 
75 and over, which has impacted wait times for 
admission to long-term-care (LTC) homes. 

The Long-Term Care Homes Act, 2007 was 
proclaimed into force in July 2010 and recog- 
nizes the principle of access based on assessed 
need and LTC homes as places where residents 
live with dignity, security, safety and comfort. 
The Ministry is pleased that the Auditor has 
recognized that the Community Care Access 
Centres (CCACs), the designated placement 
coordinator for LTC home admission, are man- 
aging various areas of the LTC home placement 
process well. 

Ontario's Action Plan for Health Care is a 
reflection of the government's commitment to 
better patient care. At the heart of the plan is 
a commitment to ensure that patients receive 
timely access to the most appropriate care in the 
most appropriate place. It is about getting the 
greatest value for patients from the system and 
ensuring seniors receive the care they need as 
close to home as possible. 

The Ministry is committed to supporting sen- 
iors to remain in their community. For example: 

• The Ontario government's Aging at Home 
Strategy, announced in August 2007, 
invested close to $1.1 billion over four years 
in the delivery of an integrated continuum 
of community-based services so seniors can 
stay healthy and live more independently in 
their homes. 

• Increased investments were made in CCACs 
to support the transition of patients from 
hospital to home or community settings. 

• In January 2011 the Assisted Living Services 
for High-Risk Seniors policy was introduced 
to address the needs of high-risk seniors who 
reside at home and require the availability 
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of personal support and homemaking ser- 
vices on a 24-hour basis to avoid premature 
admission to an LTC home. 
The The Ministry is working on launching a 
Seniors Strategy with a focus on supporting sen- 
iors to stay healthy and to stay at home longer, 
reducing the strain on hospitals and LTC homes. 
The Ministry will continue to work with the 
Local Health Integration Networks, CCACs and 
the LTC home sector to identify opportunities to 
improve the LTC home placement process. 





Detailed Audit Observa 







OVERVIEW OF PLACEMENT PROCESS 

In general, the key steps followed to place a client 
in an LTC home involve: the CCAC determining a 
client's eligibility and priority for LTC home accom- 
modation through a formal assessment process; 
clients applying to one or more LTC homes; clients 
accepted by the LTC homes being put on a wait list 
if a bed is not available; and clients at the top of a 
wait list being offered a bed. Figure 2 illustrates 
these key steps. 

INITIATIVES 

The Ministry has supported a number of initia- 
tives to help the LTC home placement process 
work consistently, fairly and in the most timely 
manner possible: 

• From 2007 to 2012, the Aging at Home 
program helped seniors stay in their homes 
longer through home-care assistance and 
thereby postponed or reduced the need for 
long-term-care accommodation. Although this 
initiative has ended, the Ministry indicated 
that the LHINs are continuing to implement 
community-based programs and services 
that support seniors, including programs and 



services designed to relieve pressures on hos- 
pitals and LTC homes by helping to find the 
appropriate health-care setting for clients. 

• In 2008, LHINs working with CCACs intro- 
duced the Wait at Home approach to provide 
CCAC-organized homemaking and personal 
support services to higher-needs clients who 
required more help than that provided in 
regular home-care hours. This enabled clients 
to wait in their homes for a long-term-care 
vacancy rather than waiting in hospital. 

• In July 2010, the Long-Term Care Homes Act 
took effect. Among other things, it introduced 
stricter eligibility criteria for LTC home place- 
ment. For example, it no longer permitted 
people access based solely on whether they 
would be financially, emotionally or physic- 
ally harmed if they stayed in their current 
residence. It also increased the number of 
LTC homes to which an individual may apply 
from three to five, and decreased the wait to 
reapply from six months to 12 weeks in cases 
where the client refuses a bed at an LTC home 
to which they applied. 

• The provincially standardized Resident 
Assessment Instrument for Home Care, which 
is used to consistently determine clients' 
eligibility for long-term care and prioritize 
clients on the basis of urgency, is continuing to 
be refined. 

• The piloting of Resource Matching and 
Referral systems, which help match hospital 
patients to the earliest available appropriate 
LTC home bed, is expected to shorten the 
placement process. At the time of our audit, 
two LHINs were testing their own systems in 
conjunction with their associated CCACs; the 
remaining LHINs were expected to pilot simi- 
lar systems during the 2013/14 fiscal year. 

• A document management system supported 
by the Ontario Association of Community 
Care Access Centres has been implemented 
by six CCACs, and the Association indicated 
that the remaining CCACs would also be 
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Figure 2: Key Steps in Long-term-care Home Placement 

Prepared by the Office of trie Auditor General of Ontario 



KEY STEPS 



I. Assessment of 
Client Eligibility 



CLIENT'S ROLE 




LTC HOME'S ROLE 



Requests eligibility 
assessment. 



Assesses whether client is 
eligible for LTC home placement 
and informs client of decision. 



II. Prioritization 
of Clients 



Determines priority level of 
eligible clients. 



III. Application 
to Home 



Completes application, 
usually choosing 1-5 homes 
in order of preference. 



Sends applications to homes 
chosen by client. 



Accepts/rejects application. 



IV. Wait for Bed 



Puts accepted client on homes' 
wait lists based on priority level 
and usually by date of 
application to home. 
Periodically reassesses client 
eligibility and priority level. 



Selects client highest on 
wait list who matches 
criteria for the bed. 



Informs CCAC that bed is 
available. 



Accepts/rejects application. 



V. Offer of Bed 



Accepts bed offer and moves^ 
into home, or rejects* offer. 



Offers bed to accepted client. 



Clients who reject a bed offer are generally removed from all LTC home wait lists, but may reapply after 12 weeks. Hospitalized clients who reject a bed offer 
stay on the LTC home wait lists, but hospitals have the option of charging these patients a hospital-determined fee to continue waiting in hospital for an LTC 
home bed. 



implementing this system. This system 
enables CCACs and LTC homes to send 
and receive clients' medical and placement 
information electronically. 
• Over the next 10 years, older LTC homes 
containing 35,000 beds will be renovated. The 
Ministry indicated that the renovations will 
provide more modern and comfortable living, 



as well as improved access, including greater 
wheelchair access for residents, and therefore 
make these homes a viable option for many 
more people. 
• A toll-free Long-term Care Action Line has 
been established to allow citizens to phone the 
Ministry with concerns and complaints about 
LTC homes and the placement process. 
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WAIT-LIST MANAGEMENT 

The number of people across the province waiting 
for an LTC home bed increased by almost 85% 
between March 2005 and March 2012, as shown 
in Figure 3, while the number of LTC home beds 
increased by about 3%. However, the number of 
people waiting decreased by almost 15% between 
March 2010 and March 2012. This was primarily 
due to the stricter eligibility criteria in the new Act. 

Of the 32,000 people on the wait list as of 
March 31, 2012, about 19,000 (or about 60%) were 
waiting for placement in an LTC home. The remain- 
ing 13,000 (or about 40%) already resided in 
long-term care, but were waiting for another, more 
preferred, home. 

On average, because residents tend to be older 
and often in poor health, they live in LTC homes for 
about three years. Therefore, although the numbers 
may vary among homes, about one-third of the 
76,000 LTC home beds in Ontario become available 
each year. 

Determining Client Eligibility 

To be eligible for an LTC home, individuals must be 
at least 18 years old and insured under the Ontario 
Health Insurance Plan. In addition, they generally 
must require the availability of 24-hour nursing 
care, supervision in a secure setting, or frequent 
assistance with activities of daily living such as 
dressing and bathing. 

Timing until Assessment 

Potentially eligible candidates come to the attention 
of CCACs in a number of ways: people may apply 
for long-term care on their own or on someone's 
behalf; the physician of a hospitalized individual 
may refer him or her to a CCAC; or CCAC staff may 
refer an existing client. 

If the individual is not an existing client, the 
CCAC generally conducts a preliminary assessment 
within a few days to determine how urgently a full 
eligibility assessment for long-term care should be 



Figure 3: People Waiting for a Long-term-care Home 
Bed, 2005-2012 

Source of data: Ministry of Health and Long-Term Care 
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Note: Data as of March 31 each year. 

* Reduction due primarily to tightened LTC home eligibility rules. 

completed. If the preliminary assessment indicates 
that the client is likely to require home care for 
more than 60 days or admission to an LTC home, 
ministry policy requires the CCAC to complete 
an eligibility assessment within the next 14 days. 
However, if the client seeking LTC home admission 
is already receiving CCAC services, there is no simi- 
lar requirement. To help reduce the time patients 
spend waiting in a hospital bed, two of the CCACs 
we visited had established stricter internal policies 
on the timeframe to assess them — three days at one 
and five days at the other — regardless of whether 
they were already receiving CCAC services. 

According to ministry information for the 
2011/12 fiscal year, province-wide the CCACs 
completed almost 80% of the assessments for hos- 
pitalized individuals within 14 days of receiving the 
request for LTC home accommodation; for people 
applying from home, 60% were assessed within 14 
days and 90% were assessed within 54 days. 

Reviewing Alternatives 

To enable people to live at home as long as possible, 
the Act requires that CCACs review all community- 
based alternatives before determining that a client 
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is eligible for long-term care. For people who do not 
need the full range of LTC home services, alterna- 
tives might include more day or respite programs, 
or supportive-housing and assisted-living options — 
for example, people live in their own apartment in a 
building that has on-site care available. 

Almost all cases we reviewed at the CCACs vis- 
ited indicated that at least some alternatives to an 
LTC home were investigated. However, satisfaction 
surveys conducted by those CCACs indicated that 
between 30% and 44% of clients did not feel they 
were informed of all the available alternatives (the 
CCACs each surveyed a relatively small number of 
LTC home placement clients). One CCAC indicated 
that many clients received CCAC care at home over 
an extended period, and that alternatives were 
explored during that time. 

According to a 2012 report by Health Qual- 
ity Ontario, a provincial agency that monitors 
and reports on health care, in the 2010/11 fiscal 
year nearly 20% of the CCACs' home-care clients 
who were subsequently placed in long-term care 
could have stayed in their homes or been placed 
elsewhere in the community. In 2011, a Ministry- 
commissioned report, Caring for Our Aging Popula- 
tion and Addressing Alternate Level of Care, indicated 
that 37% of clients waiting in hospital for an LTC 
home bed have care needs that are no more urgent 
or complex than those of many people being cared 
for successfully at home. This report also suggested 
that LTC homes should focus more of their capacity 
on restorative and transitional care programs 
(which promote, for example, the recovery of 
strength, endurance and functioning) that might 
assist clients in moving out of hospital more quickly 
and potentially returning to their own home rather 
than residing in an LTC home. However, these pro- 
grams represented only about 2% of the LTC home 
system capacity at the time of the report. 

Assessing Client Needs 

CCAC staff — generally a case manager or a place- 
ment co-ordinator — determine clients' eligibility for 



an LTC home using the provincially standardized 
Resident Assessment Instrument for Home Care 
(RAI-HC). This assesses, among other things, the 
client's level of functioning, behaviour patterns 
and requirements for personal care. It includes 
a Method for Assigning Priority Levels (MAPLe), 
which helps determine the urgency of the client's 
need for long-term care. A rating score is generated 
based on the results of the assessments. 

A client whose RAI-HC score is 7 or less is usu- 
ally not considered eligible for an LTC home, while 
a client with a score of 11 or more is considered eli- 
gible. Determining eligibility of clients with scores 
of 8 to 10 is generally based on the MAPLe score 
and the case manager's professional judgment of 
the extent of caregiver burden present in the client's 
situation. At two of the three CCACs we visited, a 
senior manager was required to review the decision 
if a client had a score of 8 to 10, and eligibility was 
often based largely on caregiver burden. The third 
CCAC had no formal secondary review process, but 
indicated that it was implementing one. 

As part of the eligibility determination process, 
the Act requires that a physician or registered nurse 
complete a health assessment. This five-page assess- 
ment provides information on, among other things, 
the client's condition, including any medications. 
The three CCACs told us that these assessments 
do not add much value to the eligibility process, as 
they are often not fully completed by the client's 
physician and duplicate information obtained 
through the RAI-HC. The Ministry indicated that 
the health assessment is an intentional secondary 
review to better ensure that all client information is 
accurate. We were informed that, in 2010, the Prov- 
incial Placement Committee (a group comprising 
representatives from each CCAC and the Ontario 
Association of CCACs) recommended discontinuing 
these health assessments. However, they were still 
required at the time of our audit. 

According to information we received from the 
Ministry, in the 2011/12 fiscal year the CCACs con- 
ducted more than 40,000 eligibility assessments of 
more than 36,000 individuals province-wide; some 
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people are assessed more than once, for instance if 
they change their minds about long-term care. The 
CCACs determined that more than 33,000 appli- 
cants were eligible for an LTC home. At the three 
CCACs we visited, the percentage of applicants who 
met the placement eligibility criteria ranged from 
88% to 93% that year. Rejected applicants may 
appeal their case to the Health Services Appeal and 
Review Board, an independent quasi-judicial tribu- 
nal established by the Ministry. The three CCACs 
informed us that appeals rarely occur. 

Determining Initial Priority Level of Clients 

In the event that accommodation is not available 
immediately, names are added to the LTC home's 
wait list based on each client's priority level. The 
Act stipulates the priority levels and the eligibil- 
ity for each level. The CCACs are responsible for 
determining at which priority level clients should 
be placed. The Ministry and CCACs told us that the 
highest priority levels are seldom used — only 33 
such clients were on the wait list as of March 31, 
2012. These priority levels include a category 
for veterans (who are eligible for veterans' beds, 
which are less than 1% of all LTC home beds), and 
a category for clients requiring readmission to an 
LTC home after being discharged involuntarily, 
such as following an extended hospitalization. 
Figure 4 outlines the other, more commonly used 
priority levels. 

Not all the priority levels are based exclusively 
on medical need — for example, the federally sup- 
ported veterans' beds in LTC homes are available 
to veterans, even if they have no health-care needs 
or would not otherwise be eligible. These beds 
therefore may not be available for clients with care 
needs. Similarly, under priority levels 3B and 4B, 
spouses of current residents wishing to live in the 
same LTC home do not require any care needs to 
be eligible. And, while category 2 clients must have 
care needs, these clients are prioritized within level 
2 based on their spouse/partner's date of admission 
to the home, even though their care needs may not 



be as high as others'. Therefore, in many of these 
cases, clients with lesser care needs may be ranked 
ahead of clients with higher care needs. 

If a person's condition deteriorates, he or she 
may be re-prioritized to a higher level. However, cli- 
ents will be placed on the 3A and 4A wait list using 
their original 3B or 4B wait-list date — and therefore 
possibly ahead of other 3A or 4A clients who have 
been waiting at a higher-needs priority level for a 
longer time. 

All three of the CCACs used judgment in deter- 
mining which clients were a crisis priority, includ- 
ing a determination of the caregiver's burden. 
Caregiver burden considerations can potentially 
give one client priority over another whose needs 
are more urgent and whose caregiver burden is at 
least as arduous but whose caregivers are less insist- 
ent. To reduce the risk of inappropriately designat- 
ing clients as crisis, one CCAC required a senior 
manager to review each crisis designation and 
sign a crisis approval form. In most of the files we 
reviewed, the senior manager had signed this form. 
Another CCAC required a second case manager to 
review each crisis designation. However, 50% of 
the files we reviewed did not indicate whether this 
review had been completed. The third CCAC did 
not require crisis designations to be reviewed by a 
second person, but said that more borderline cases 
could be discussed between case managers. 

We noted that two of the CCACs used additional 
factors, which were not specifically based on needs, 
for designating clients as crisis. For example, one 
CCACs policy included designating clients as crisis 
if they had waited three years in an LTC home that 
was not their first choice. Another CCAC, which had 
a program to enable hospitalized clients to wait at 
home for an LTC placement, designated such clients 
as crisis once they had waited at home for 30 days, 
mostly because of difficulties the clients had with 
coping at home. 

Patients waiting in hospital for an LTC home 
bed are generally prioritized as a 3A or 4A, with 
no priority over people waiting in the community. 
We noted that one other province gives a higher 
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Figure 4: Commonly Used Long-term-care Home Priority Levels 

Source of data: Long-Term Care Homes Act and Ministry of Health and Long-Term Care 
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this home, but this is not their first choice of homes; or 

• does not have care needs, but wishes to reside with 
spouse/partner who is already in the home 


By date of application to 
LTC home 


Other clients 


4A 


Client not eligible for any other higher priority level who: 

• has higher care needs*; 

• is waiting in hospital; or 

• is residing in another LTC home, but this is their first choice 
of LTC homes 


By date of application to 
LTC home 




4B 


Client not eligible for any other higher priority level who: 
• is eligible based on care needs; 


By date of application to 
LTC home 



• is residing in another LTC home and has applied for a bed in 
this home, but this is not their first choice of homes; or 

• does not have care needs, but wishes to reside with 
spouse/partner who is already in the home 



* The Provincial Placement Committee's guideline indicates that clients with higher care needs are generally those with a RAI-HC score of 16 or higher, or those 
with both a RAI-HC score of 11 to 15 and a MAPLe score of 4 or higher. 



wait-list priority to people who are waiting in the 
hospital, in order to more quickly free up hospital 
beds for other patients. In Ontario, in order to free 
up hospital beds more quickly when a hospital is 
experiencing severe capacity pressures, the LHIN 
can declare the hospital to be "in crisis," and all 



patients waiting for an LTC home in this particular 
hospital are generally given crisis priority. When 
these patients move up to the crisis priority level, it 
causes other 3A/4A patients, both in other hospitals 
as well as in the community, to wait longer for an 
LTC home. In 2011, two of the CCACs we visited 
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had a combined total of nine hospitals declared in 
crisis; and one of the hospitals was designated as 
being in crisis for most of that year. 

The Act requires that crisis clients be prioritized 
on the basis of urgency of need. However, the 
CCACs we visited told us that all crisis clients have 
high needs, so it is often hard to distinguish whose 
needs are more urgent. 

One of the CCACs prioritized crisis clients on 
the basis of their total wait time for an LTC home. 
Another CCAC prioritized most crisis clients on 
their total wait time as well, but also maintained 
a "high-crisis" list that gave top priority to clients 
with the most urgent needs waiting in the com- 
munity. The third CCAC generally gave priority to 
crisis clients according to their wait time in just the 
crisis category. 

After crisis clients, spouses and partners — 
including relatives or friends — of current LTC home 
clients are the next-highest-level priority. During 
our audit, the Ministry clarified its definition of 
partner to include only those individuals who had 
lived with the client during the year preceding 
the client's application to an LTC home. All three 
CCACs visited had adopted this clarified definition. 

The Act gives clients seeking LTC homes that 
serve their religion, ethnic origin or linguistic origin 
a category 3 priority level. Although no documenta- 
tion is required to be placed in this priority level, 
it is important for CCACs to accurately identify 
clients who are entitled to this priority. However, 
for three of the specialty homes at one CCAC, 26 
clients, accounting for 75% of all the clients ranked 
at the priority level 4A/4B, actually qualified for the 
higher 3A/3B priority level. We brought these cases 
to the attention of the CCAC, which reclassified 
them to priority level 3A/3B. While all the clients 
moved up the wait list, nine of the clients moved up 
the wait list by more than 600 people. This CCAC 
indicated that, because no one moved to the top 
position on the wait list, none of these clients had 
missed a bed offer. 



Placing Clients on the Wait List 

Client Application to Homes 

The Act requires CCACs to provide clients with 
information on the implications of different LTC 
home choices, and, if the client wishes, assistance 
in selecting homes. Clients eligible for an LTC 
home generally select a maximum of five homes, 
with crisis clients permitted to select an unlimited 
number of homes. Clients complete an application 
for the home(s) to which they wish to apply; if 
more than one is selected, they rank them in order 
of preference. This application is provided to the 
CCAC. However, there is no deadline for completing 
the application. This gives clients and their family 
time to consider their options carefully. However, 
for hospital patients it can also extend the time that 
they occupy a hospital bed. 

Under the Act, clients have the right to voluntar- 
ily choose which LTC home(s) they want to apply 
to; in February 2011, and again in May 2012, the 
Ministry clarified with the LHINs that clients can- 
not be required to choose from a pre-selected list of 
homes. In essence, clients can only be placed in a 
home that is acceptable to them. However, we noted 
that one CCAC had a policy of asking crisis clients to 
select homes with current vacancies or short waiting 
lists if the applicant's selected home(s) could not 
accommodate immediate admission. If the client did 
not agree to do this — perhaps because of distance 
from family or because the homes were older facili- 
ties — he or she might lose the crisis designation, and 
be moved to a lower priority level by the CCAC. We 
noted that another CCAC had a policy in place until 
October 2011 that required crisis clients to choose 
all homes within 70 kilometres of their residence, 
or similarly risk losing their crisis designation. 
However, the Ministry required this CCAC to change 
the policy, stating that the crisis designation is based 
on the clients' condition or circumstances, not on 
their willingness to consider alternative LTC home 
choices. This CCAC indicated that it focuses on cli- 
ents requiring immediate placement when designat- 
ing clients as crisis. 



2012 Annual Report of the Office of the Auditor General of Ontario 



Numerous studies have shown that remaining 
in hospital longer than medically necessary, includ- 
ing waiting in hospital for an LTC home, can be 
detrimental to a person's health for various reasons, 
among them the potential for a hospital-acquired 
infection such as C. difficile, and, for older patients, 
a decline in physical and mental abilities due to 
lack of activity. As well, it is much more costly for a 
person to wait in hospital than in an LTC home or 
at home with appropriate home-care support, and 
it might prevent another person requiring hospital 
care from occupying that bed. We determined using 
ministry data that, during the 2011/12 fiscal year, 
19% of clients waiting in hospital had applied to 
only one home. In fact, at one CCAC we visited, 
35% of clients waiting in hospital had applied to 
only one LTC home. While this practice would tend 
to increase the time clients wait in hospital for a bed 
in their preferred home, we noted that the median 
wait time province-wide for 3A and 4A hospitalized 
clients was about half that of people waiting in the 
community. This may be due to hospitals encour- 
aging clients to apply to homes with vacant beds. 

Rather than allowing patients to wait in 
hospital for their preferred home, our research 
indicated that many other provinces have stricter 
policies: five provinces require the patient to go 
to the first vacant bed in any LTC home; and two 
provinces require patients to go to any LTC home 
within 60 and 100 kilometres, generally of their 
home, respectively. 

Acceptance/Rejection by LTC Homes 

The CCAC forwards client applications, including 
information on the client's care needs, to the applic- 
able LTC homes. At the three CCACs visited, this 
information was usually faxed. At the time of our 
audit, the Ontario Association for Community Care 
Access Centres was piloting a system at six CCACs 
for the electronic transmission of documents to and 
from LTC homes. 

Under the Act, LTC homes generally have five 
business days to accept or reject an application. 



If the home requires more information from the 
CCAC, the home has three additional days after 
it receives the information to make its decision. 
According to ministry data, in the 2011/12 fiscal 
year LTC homes province-wide made a decision 
on 65% of the applications within five business 
days. LTC homes responded to 90% of applications 
within 28 days. 

We were informed that LTC homes rarely reject 
clients unless they have very high care needs; in the 
2011/12 fiscal year, only about 1% of clients' appli- 
cations were rejected. An accepted client can move 
into the LTC home immediately if a bed is available. 
However, in most cases the client is added to the 
home's wait list based on the client's priority level 
because no bed is currently available. 

Reassessing Clients 

In some cases a client's condition can deteriorate 
significantly while waiting for an LTC home bed. 
These changes might merit adjusting the client's 
priority level to a higher level, such as a crisis 
priority. As well, when beds become available, LTC 
homes require up-to-date information about the 
care needs of the clients who are moving in. 

CCACs may be made aware of changes in a 
client's condition by various means, for instance 
when the client's family contacts the CCAC or 
when the CCAC conducts a reassessment. The Act 
requires that clients have an assessment or reassess- 
ment within three months of their placement in 
long-term care, which helps ensure LTC homes 
have up-to-date information to prepare for the 
client's needs. This involves completing all aspects 
of the initial eligibility assessment again, including 
the RAI-HC assessment and the health assessment 
(generally obtained from the client's physician). As 
well, ministry policy states that a reassessment of 
the RAI-HC should be completed every six months. 
This applies to clients requiring care at home, to 
ensure their care plans meet their needs, as well as 
to clients not receiving care at home and hospital- 
ized clients. CCACs province-wide completed a 
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combined total of about 36,000 reassessments 
of clients on LTC home wait lists in the 2011/12 
fiscal year. 

The CCACs visited confirmed that it is chal- 
lenging to complete all required reassessments, and 
therefore they used their resources as follows: 

• Two CCACs ensured that assessments/ 
reassessments for the 10 highest-priority 
clients on each home's wait list had been 
completed in the past three months. As well, 
one CCAC indicated that clients with condi- 
tions that were likely to change would be 
reassessed every six months, while the other 
CCAC indicated that it conducted as many 
six-month reassessments as possible. 

• The third CCACs approach was to reassess 
3A/4A clients every three months and 3B/4B 
clients every six months. (Clients ranked 
3A/4A at each home are eligible for beds 
before 3B/4B clients.) 

For hospitalized clients and clients not receiving 
CCAC care at home, conducting a quick "touch- 
base" with clients and other appropriate persons, 
including families, could more quickly provide 
the CCAC with information on whether a client's 
condition has changed, and therefore whether a 
more formal reassessment is needed to determine, 
for example, whether a client's priority level might 
have also changed. Furthermore, identifying and 
reassessing clients who are likely to be placed 
within the next three months, such as those nearing 
the top of the wait lists, might provide LTC homes 
with the information needed to prepare for clients 
without requiring that the CCAC repeatedly con- 
duct a formal reassessment of all individuals every 
six months. 

As well, a quick "touch-base" might identify 
clients already in an LTC home who stayed on wait 
lists for other more preferred homes, but who have 
subsequently decided to remain where they are. 
One CCAC indicated that it followed up with clients 
after six weeks, and all three CCACs indicated 
that they followed up annually, for example, as 
part of client reassessments. However, one CCAC 



visited did a one-time check of clients already in 
an LTC home who were on the wait list for another 
home, resulting in about 10% of those clients being 
removed from the wait lists. 

Placing Clients in LTC Homes 

When a bed becomes available, the LTC home noti- 
fies the CCAC and provides information on the type 
of bed. This information includes whether the bed 
is basic (varies from one or two people per room 
in newer homes, to three or four people in older 
homes) or preferred (that is, semi-private or pri- 
vate) to match the client's request; whether a bed in 
a shared room is appropriate for a male or female; 
and whether the bed is in a locked area and appro- 
priate for clients requiring secure accommodation. 

Once notified, the CCAC selects the client at 
the top of that home's wait list who matches the 
specifications of the available bed. To better match 
hospital patients to the first available appropriate 
LTC home bed, Resource Matching and Referral 
systems were being piloted in two LHINs, and the 
remaining LHINs were expected to pilot similar sys- 
tems during the 2013/14 fiscal year. The CCAC then 
sends current information on the selected client to 
the LTC home. The LTC home reviews the informa- 
tion and may accept or reject the client. Rejections 
from LTC homes at this stage are generally because 
the client's needs have changed significantly since 
the home accepted the initial application. If the 
applicant is rejected, the client is notified and the 
CCAC generally removes this client from the home's 
wait list and proceeds to the next person. Rejected 
clients may apply to another home. 

If the client is accepted, the CCAC contacts the 
individual and offers him or her the bed. Under 
the Act, the client has 24 hours to respond. If the 
client accepts the bed offer, the Act requires that 
he or she move into the LTC home within five days. 
Our review of ministry data indicated that 83% of 
beds offered to clients were accepted province-wide 
in the 2011/12 fiscal year. Furthermore, 36% of 
the clients who were placed got their first choice 
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of homes. Other people who had not selected the 
home as their first choice nonetheless were offered 
the bed, and, after accepting, generally remained 
on the wait list for their preferred home(s). 

If a client applying from the community rejects 
a bed, he or she is generally removed from all LTC 
home wait lists for a period of 12 weeks. If the 
client was waiting in hospital, he or she may remain 
on the wait list for LTC homes, but under the Public 
Hospitals Act the hospital has the option of charging 
the client a hospital-determined fee to continue 
waiting in a hospital bed. 

In the latter half of the 2010/11 fiscal year, two 
of the CCACs visited incorporated a process into 
their information system to assist in tracking the 
status of available LTC home beds, including when 
the bed became available, to whom the bed was 
offered, the date the bed was offered, and whether 
the client accepted or rejected the bed. The third 
CCAC used spreadsheets to track this information 
but indicated that it was implementing an inte- 
grated system similar to that of the other CCACs. 

We noted that CCACs did not periodically review 
client placement decisions in order to ensure the 
highest-priority person was offered the available 
bed. Furthermore, the information system used by 
the CCACs did not have the capability to retrieve 
what an LTC home's wait list looked like on a 
specific date, and therefore CCACs could not review 
these decisions after the fact. 



RECOMMENDATION 1 



To better ensure that higher-needs clients are 
identified and placed in long-term-care homes 
(LTC homes) as soon as possible, Community 
Care Access Centres (CCACs) should: 

• develop a consistent province-wide pro- 
cess for ranking clients within the crisis 
priority level; 

• in consultation with the Ministry, consider 
conducting a periodic "touch-base" to deter- 
mine whether wait-listed clients' condition 
or circumstances have changed and there- 



fore require a reassessment of their needs, 
rather than conducting formal reassessments 
of all clients every six months as is currently 
required; and 
• conduct periodic independent reviews of 
placement decisions to ensure that the 
highest-priority client matching the bed 
specifications (such as male versus female, 
and private versus semi-private and basic 
accommodation) is offered the first available 
LTC home bed. 



All of the CCACs visited agreed with this recom- 
mendation and indicated that they would: 

• work with the Provincial Placement Working 
Group (a group comprising representatives 
from each CCAC and the Ontario Association 
of CCACs) to develop a consistent province- 
wide process for ranking clients within the 
crisis priority level; 

• work with the Ministry of Health and 
Long-Term Care to create a province-wide 
standardization of the current CCAC pro- 
cesses to include a periodic "touch-base" 
to determine whether wait-listed clients' 
conditions or circumstances have changed 
and whether a formal reassessment is 
required; and 

• in conjunction with the Ontario Association 
of CCACs, develop the necessary reports 

to conduct periodic independent reviews 
of placement decisions to ensure that the 
highest-priority client matching the bed 
specifications is offered the available LTC 
home bed. 



RECOMMENDATION 2 



To help clients move out of hospital more 
quickly and to help manage growing wait lists, 
the Ministry of Health and Long-Term Care 
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(Ministry) should consider options employed 
by other jurisdictions, as well as making more 
community alternatives to long-term-care 
(LTC) homes available and having LTC homes 
provide more restorative and transitional care 
programs to improve, among other things, 
clients' functioning. 

As well, to better ensure that clients assessed 
as eligible for an LTC home are placed as soon 
as possible, the Ministry should streamline the 
client health assessment (to avoid duplicating 
information that is already obtained as part of 
the eligibility assessment and to avoid poten- 
tially delaying the process) . 



| MINISTRY RESPONSE 




The Ministry values the Auditor General's rec- 
ommendation and will continue to demonstrate 
its commitment to supporting seniors to remain 
in their community through more community 
alternatives. For example: 

• As part of Ontario's Action Plan for Health 
Care the Ministry has announced a Seniors 
Strategy focusing on supporting seniors to 
stay healthy and live at home longer through 
enhanced preventative care and home-care 
services, thereby reducing pressures on 
LTC homes and hospitals. The strategy will 
help inform decisions regarding the role of 
restorative and short-stay programs in LTC 
homes and the future development of com- 
munity alternatives closer to home. 

• The 2012/13 Community Sector Invest- 
ment in home care and community services 
announced in the 2012 Ontario Budget 
was allocated to the Local Health Integra- 
tion Networks (LHINs) in August 2012 to 
increase investments in home care and 
community services to support seniors and 
other Ontarians at home and to reduce the 
numbers of emergency room visits, patients 
waiting in hospital for an alternative level of 



care, and avoidable hospital readmissions. 
The government also signalled its intent to 
increase financial support to the community 
sector for three fiscal years. 
• Regulatory amendments made in June 2008 
and September 2009 enabled innovation 
and flexibility in the delivery of Community 
Care Access Centre (CCAC) home-care and 
community services. Service maximums for 
personal support/homemaking and nursing 
were increased, and new services and service 
locations were introduced. 
The current health assessment requirements 
were a response to concerns relating to the 
content and timing of assessments during the 
placement process that were identified as a 
result of a Coroner's Inquest into the deaths of 
two residents at the hands of another resident 
on his first day of admission (Casa Verde Nurs- 
ing Home). In some instances duplication is a 
necessary part of verifying information about 
a client's status and is a critical component of a 
comprehensive assessment. The Ministry will 
work with the CCACs and LTC homes to review 
the health assessment process and will look for 
opportunities to reduce unnecessary duplication 
in the process. 



WAIT TIMES 

According to ministry data, 50% of clients 
province-wide (excluding crisis, spousal/partner 
reunifications and persons waiting for a transfer 
to another home) were placed in an LTC home 
within 98 days in the 2011/12 fiscal year, with 75% 
of clients placed within 10 months and 90% of 
clients placed within about two years. This reflects 
the wait from the time a CCAC received a client's 
request to be assessed for an LTC home until the 
client was placed in a home. The median wait for all 
clients not yet in an LTC home, including crisis and 
spousal/partner reunification, drops to 85 days, 
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primarily due to 90% of crisis clients being placed 
within three months of being designated as crisis. 
During the 2011/12 fiscal year, 15% of clients died 
before receiving LTC home accommodation. 



in 2021, the first of the baby boomer generation — 
those born between 1946 and 1964 — will start to 
turn 75, at which point the demand for long-term 
care is expected to become even greater. 



Wait Time Trends 



Wait Times by CCAC and Client Priority 



As Figure 5 shows, the median wait times have 
almost tripled from 36 days in the 2004/05 fis- 
cal year to 98 days in the 2011/12 fiscal year. An 
increase in the number of LTC home beds of 3% 
during that period has not kept pace with the rising 
demand from an aging population. However, the 
wait time has decreased since 2009/10, due in part 
to tighter eligibility requirements under the new 
Act that took effect in July 2010. 

About 85% of LTC home residents are aged 75 
and over, and between 2005 and 2012 the number 
of Ontarians aged 75 and older increased by more 
than 20%. According to Statistics Canada, between 
2012 and 2021, Ontario's population aged 75 and 
older is expected to increase by almost 30%. This 
trend will likely increase the demand for long-term 
care, although enhanced community alternatives 
could meet some of these needs. As well, beginning 

Figure 5: Long-term-care Home Median Wait Times* 
and the Number of Ontarians Aged 75 and Older, 
2004/05-2011/12 

Source of data: Ministry of Health and Long-Term Care and Statistics Canada 
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* Excludes crisis, spousal-partner reunifications and transfers between 
LTC homes. 



Ministry data indicates that applicants who live 
in some areas of the province get into LTC homes 
more quickly. In particular, median wait times for 
LTC homes in the 2011/12 fiscal year ranged from a 
low of 50 days at the Erie St. Clair CCAC to a high of 
187 days at the North West CCAC. (Overall, 90% of 
clients were placed within a low of 317 days at the 
Central West CCAC to within a high of about 1,100 
days at the Champlain CCAC, as shown in Figure 6.) 
Part of this variance reflects differences among 
CCACs in the demand for long-term care due to 
population health and age characteristics in that 
region of the province, as well as the number, age 
and location of LTC home beds. 

According to the Ontario Hospital Association, 
as of March 31, 2012, about 1,000 people were 
waiting province-wide in an acute-care hospital 
bed for a bed in an LTC home; another 1,000 were 
waiting in other types of hospital beds, such as 
rehabilitation or mental health beds. We noted that 
the number of people waiting in hospital for an LTC 
home bed had decreased by about 25% since March 
2010. However, for those still waiting, the wait time 
until placement had significantly increased. Min- 
istry information indicated that, in the 2011/12 fis- 
cal year, about half of the acute-care patients were 
placed within about two months (within one month 
in the 2009/10 fiscal year), with 90% placed within 
495 days (within 128 days in 2009/10). Wait times 
in hospital tend to be longer for harder-to-care-for 
patients, for example, people who have dementia, 
are significantly overweight, or require frequent 
medical treatments such as dialysis. 

The Act requires CCACs to provide an "estimated 
wait time" to clients who request information on 
their expected wait. The CCACs visited said that 
clients have become upset if their actual wait time 
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Figure 6: Number of Days Within Which 90% of Each CCAC's Clients Were Placed, 2011 

Source of data: Ministry of Health and Long-Term Care 
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exceeded the estimate. Therefore, the CCACs gener- 
ally tell clients how long the top person on the wait 
list, at the same priority level as the client, has been 
waiting so far. We reviewed ministry wait-list data as 
of March 31, 2012, to see how long clients had been 
waiting. For those in the crisis category, including 
those who previously waited at a lower priority level, 
their median total wait time up to that point was 
94 days. People in categories 3A and 4A had been 
waiting a median of 423 and 309 days, respectively, 
with people in categories 3B and 4B (considered to 
have lower needs than 3A and 4A clients) waiting a 
median of 712 and 587 days, respectively. 

In the 2011/12 fiscal year, about 5,600 (or 
22%) of the LTC home placements were for clients 
in the crisis priority level and another 71% were 
from the generally higher-need categories 3A and 
4A. As shown in Figure 7, based on the number of 
people waiting and the number actually placed in 
the prior year, the expected time to placement for 



people in the lower priority levels could often be 
many years. 

As well, the high priority given to crisis clients 
may make it difficult for non-crisis clients to get 
into some homes. For example, ministry data shows 
that at more than 70 LTC homes, at least half of 
admissions during the 2011/12 fiscal year were 
crisis clients. In fact, at one CCAC visited, crisis cli- 
ents made up two-thirds of placements to the four 
most popular LTC homes — that is, the homes with 
the longest wait lists. Furthermore, ministry data 
showed that more than 40% of crisis placements in 
the 2011/12 fiscal year were people whose prior- 
ity level was escalated to crisis (for example, due 
to their condition deteriorating or circumstances 
changing, such as increased caregiver burden) in 
order to place them quickly after they had waited at 
a lower priority level, in some cases for an extended 
period of time. 



2012 Annual Report of the Office of the Auditor General of Ontario 



Figure 7: Number of Clients Waiting for Long-term Care, Number of Placements and Expected Average Time to 
Placement for Commonly Used Priority Levels 

Prepared by the Office of the Auditor General of Ontario 
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3A - Religious, ethnic, linguistic home (higher needs) 
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Impact of Client Choices on Wait Times 

Although the Ministry provides funding to LTC 
homes, residents must make a co-payment for their 
accommodation costs. As of July 2012, the monthly 
co-payment ranged from a low of almost $1,700 per 
month for basic accommodation to a high of over 
$2,200 for private accommodation. The co-payment 
for semi-private and private accommodation in 
newer homes (that is, the approximately 55% of 
long-term-care beds that generally meet or exceed 
the Ministry's 1999 LTC home design standards) is 
about $30 to $50 more respectively per month than 
for older homes. Financial assistance is available 
from the Ministry if a resident is unable to pay, but 
only for basic accommodation. 

The Act allows LTC homes to designate up to 
60% of their beds as preferred — private or semi- 
private — accommodation, which means a minimum 
of 40% of their beds must be basic accommodation. 
However, almost 60% of clients applying for an LTC 
home bed requested basic accommodation. There- 
fore, clients who can afford to pay for preferred 
accommodation may get placed faster than clients 
applying for basic accommodation, regardless of 
their medical needs. 

Furthermore, people pay the same rate for basic 
accommodation in homes of any age, even though 
basic accommodation in a newer home has only one 



or two people per room, compared with up to four 
people per room in an older home. Partially as a 
result of this, newer homes tend to have longer wait 
lists (many over 1,000 people), while less desirable 
homes may have empty beds for lengthier periods 
of time. Therefore, clients selecting less desirable 
LTC homes that have available beds or short wait 
lists can get placed more quickly. 

Public Reporting of Wait Times 

Health Quality Ontario publishes the overall prov- 
incial LTC home wait time annually. However, no 
information was reported on regional wait times or, 
more specifically, the wait times for each LTC home. 
Reporting wait times for particular homes is some- 
what complex because wait times vary based on a 
number of factors, including the priority level of the 
client and the type of accommodation chosen. 

In August 2012, one of the CCACs we visited 
began publicly disclosing on its website informa- 
tion on wait times for each LTC home in its region, 
including the number of clients waiting for each 
type of accommodation, the average number of 
days that clients have waited so far, and the average 
number of beds that became available monthly. We 
believe this is a good initiative for providing public 
wait time information. 
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RECOMMENDATION 3 



To better ensure that clients have sufficient 
information on the long-term-care (LTC) home 
placement process and wait times for LTC home 
admission, the Ministry of Health and Long-Term 
Care (Ministry), in conjunction with the Com- 
munity Care Access Centres (CCACs), should: 

• provide the public with detailed information 
on the LTC home admission process and 
the policies in place to ensure the process is 
administered equitably; 

• examine options for encouraging greater 
utilization of basic accommodation in less 
desirable homes; and 

• promote the public disclosure of informa- 
tion that would help people choose which 
LTC homes to apply to, such as wait times 
by home, by type of accommodation — pri- 
vate, semi-private and basic — as provided 
on one CCACs website, and wait time by 
priority level. 

| MINISTRY RESPONSE 

The Ministry agrees with the principles identi- 
fied in the recommendation and will work with 
the CCACs, through the Ontario Association of 
Community Care Access Centres, to continue 
to promote ongoing and timely communication 
of appropriate information regarding the LTC 
home placement process. The Ministry will con- 
tinue to work in partnership with CCACs in the 
ongoing review of these policies and processes. 
Further, the Ministry regularly reviews the poli- 
cies and processes to ensure maximum utiliza- 
tion of all levels of accommodation in all homes. 

AC RESPONSI 

Although this recommendation was not directed 
toward the CCACs, one CCAC highlighted its sup- 
port for the disclosure of information that would 
help people choose which long-term-care homes 
to apply to, such as average wait times by home. 



OVERSIGHT 

The Ministry is responsible for ensuring compliance 
with the Long-Term Care Homes Act (Act), which 
includes the LTC home placement process. The 
Ministry indicated it commenced monitoring the 
placement process in fall 2011, primarily by follow- 
ing up on complaints it received directly. We noted 
that the three CCACs visited all had processes in 
place for handling complaints. They indicated that 
very few complaints — on average, fewer than 20 
per year — had been received about their LTC home 
placement process. Although one CCAC provided 
some general complaint statistics to its LHIN, no 
other information on complaints was regularly 
provided by the CCACs to either the LHINs or 
the Ministry. 

Otherwise, the Ministry has delegated the 
oversight of the CCACs to the LHINs. The Local 
Health System Integration Act restricts LHINs' access 
to information on individual clients. However, 
through the LHINs' performance agreements with 
the CCACs, the LHINs receive information on the 
percentage of higher-needs clients who are placed 
in LTC homes. Furthermore, the CCACs visited all 
provided additional information to their LHINs 
regarding the LTC home placement process, such 
as the number of crisis placements by location in 
which the client was waiting and the number of 
placements by hospital. 

Although the Ministry agreed with the recom- 
mendation in our 2010 Discharge of Hospital 
Patients report, regarding the need to establish 
benchmark standards for completing each stage in 
the LTC home placement process (such as times to 
determine client eligibility, for hospital clients to 
complete applications, to get clients onto a wait list, 
and to place clients), there are still few benchmark 
standards for this process. (See the Discharge of 
Hospital Patients follow-up section in Chapter 4 for 
additional information on the current status of this 
recommendation.) Other performance measures 
that would provide the Ministry or LHINs with 
information about the effectiveness of the LTC 
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placement process could include wait times for 
clients requesting preferred versus basic accom- 
modation; wait times for clients waiting in hospital 
versus at home; percentage of clients who die while 
awaiting placement; and percentage of clients 
who receive their requested transfer to another 
LTC home. 



RECOMMENDATION 4 



To enhance the oversight of the long- term-care 
(LTC) home placement process, the Ministry 
of Health and Long-Term Care (Ministry), in 
conjunction with the Local Health Integration 
Networks (LHINs) and Community Care Access 
Centres (CCACs), should: 
• develop consistent performance measures 
for monitoring the process, such as wait 
times for clients waiting in hospital versus 



at home, wait times for clients requesting 
preferred (that is, private or semi-private) 
versus basic accommodation, and the 
percentage of clients who receive their 
requested transfer to another LTC home; and 
develop target guidelines for completing 
each stage of the LTC home placement pro- 
cess, such as the times to determine client 
eligibility, for hospital clients to complete 
placement applications, and for clients to get 
onto a wait list. 



The Ministry will continue to review the 
data requirements as necessary to provide 
regular reporting on and monitoring of 
performance measures. 



Chapter 3 
Section 




Metrolinx— Regional 
Transportation Planning 



Background 




Metrolinx, an agency of the government of Ontario, 
was created by the Greater Toronto Transportation 
Authority Act, 2006, now the Metrolinx Act, 2006 
(Act) . According to the Act, one of Metrolinx's 
key objectives is to provide leadership in the co- 
ordination, planning, financing and development of 
an integrated, multi-modal transportation network 
in the Greater Toronto and Hamilton Area (GTHA) . 
The GTHA consists of two single-tier municipalities 
(Toronto and Hamilton), four regional munici- 
palities (Durham, Halton, Peel and York) and 24 
local municipalities. 

In November 2008, Metrolinx formally adopted 
a Regional Transportation Plan (RTP) — also known 
as "The Big Move" — that sets out the priorities, 
policies and programs for implementing a trans- 
portation system within the GTHA. The RTP, which 
was the result of two years of public consultation, 
was adopted by Metrolinx's Board of Directors 
(Board), which at that time included representa- 
tives from the GTHA municipalities. 

Among the RTFs more significant proposals is 
to build more than 1,200 km of rapid transit with 
the aim of getting 80% of GTHA residents within 
2 km of rapid transit. The timeline for implementing 
the RTP is 25 years. Its estimated cost of $50 billion 



relates only to upgrading and expanding the regional 
transportation network but does not include the esti- 
mated maintenance that is expected to be required 
to keep the additional transportation infrastructure 
in a state of good repair over its useful life. 

In the first 15 years, Metrolinx plans to imple- 
ment the priority transit projects listed in Figure 1. 
Metrolinx's estimate of the cost of these projects is 
approximately $33 billion, of which approximately 
$3 billion had been spent by the province as of 
March 31, 2012. For about half of these projects, 
the majority of the funding comes from a 2007 
provincial commitment of $11.5 billion, along with 
previously announced project funding. The remain- 
ing priority projects that are funded — such as the 
Air Rail Link between Union Station and Pearson 
International Airport and projects to revitalize 
Union Station — are being funded from the prov- 
ince's capital budget for GO Transit (the commuter 
rail and bus system serving the GTHA, a division of 
Metrolinx) . At the time it made the 2007 commit- 
ment, the province asked the federal government to 
contribute $6 billion toward the RTP's implementa- 
tion. To date, the federal government has com- 
mitted $1.93 billion on a project-by-project basis. 
The combined funding is expected to sustain the 
RTP's implementation until about 2018. By 2013, 
Metrolinx must provide the province with recom- 
mendations for funding the implementation of the 
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Figure 1: List of Priority Transit Projects in the Regional Transportation Plan's First 15 Years 

Source of data: Metrolinx 









Provincial 




Estimated 1 
Capital Cost 




Spending as of 
March 31, 2012 


Transit Priorities 


($ million) 


Funded 


($ million) 


Fxnrp*^ rail ^prvirp frnm Hamilton tn ftehawa 

LAU 1 vOO 1 CI 1 1 vJV^ 1 V 1 v/V-/ 11 \J II 1 1 1 LA 1 1 1 1 1 LVJ 1 1 LU WJI IU VVU 


5,970 


No 2 




Ranirl transit linp in Hnwntnwn Hamilton 
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830 


No 2 
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No 2 
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259 


Yes 3 


89 


Hnrnntarin ranirl transit frnm Pnrt Hrprlit tn rlnwntnwn Rramntnn 
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1,350 


No 2 




Brampton's Queen Street AcceleRide (now Zum) 


259 


Yes 3 


95 


Air Rail Link (ARL) between Union Station and Pearson Airport 4 


456 


Yes 3 


40 


Upgrades to Georgetown South line 


1,501 


Yes 3 


407 


VIVA bus rapid transit on Highway 7 and Yonge Street through York Region 


1,755 


Yes 3 


295 


Yonge Street subway capacity improvements and extension to Richmond Hill 


2,380 


No 2 




Spadina subway extension to Vaughan Metropolitan Centre 


2,600 


Yes 3 


891 


Rapid transit on Eglinton Avenue 


4,600 


Yes 3 




Finch/Sheppard rapid transit 


2,150 


Yes 3 


471 


Upgrade and extension of Scarborough rapid transit line 


1,400 


Yes 3 




Rapid transit service along Highway 2 in Durham 


500 


No 2 




Improvements to existing GO rail services and extension of GO rail service to 
Bowmanville 


4,300 


No 2 




Other Projects 


Presto fare card 


701 


Yes 3 


364 


Union Station revitalization (a combination of projects) 


1,393 


Yes 3 


355 


Total 


33,054 




3,007 



1. Estimated capital costs were established in 2008, 2009 or 2010 depending on the project. 

2. Funding is pending, so work has not yet been started on these projects. 

3. Funding is in place for these projects, so in most cases work has begun. 

4. The ARL will benefit from the upgrades to the Georgetown South line (see following row, beneath dotted line). 



remaining projects contemplated under the RTP's 
first 15 years as well as the projects contemplated in 
years 16 through 25. 



Audit Objective and Scope 



The objective of our audit was to assess whether 
Metrolinx had adequate systems and procedures in 
place to: 

• cost-effectively implement the initial stages of 
the Regional Transportation Plan (RTP) ; and 



• regularly report on activities and progress 
toward achieving the RTP. 

Senior management of Metrolinx reviewed 
and agreed to our objective and associated 
audit criteria. 

We looked at the delivery to date of three major 
capital projects contemplated within the RTP's 
first 15 years, whose construction or development 
was under way at the time of our audit — the Air 
Rail Link, the Presto fare card and two significant 
projects that form part of the Union Station revital- 
ization (restoring the train shed and replacing the 
switches in the Union Station Rail Corridor) . 
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Our audit also assessed the processes followed 
and progress made in implementing the RTP since 
its adoption in 2008. 

The audit was primarily conducted at Metro- 
linx's head office in Toronto, where we interviewed 
staff and reviewed pertinent documents. We also 
interviewed representatives from many of the 
regions and municipalities within the GTHA, along 
with representatives of their respective transit agen- 
cies, to obtain their perspective on various aspects 
of the RTP, its overall implementation and the indi- 
vidual projects currently being implemented within 
the plan. As well, we researched transportation 
planning in other jurisdictions similar to the GTHA 
to identify best practices and lessons learned that 
could be applied to implementing the RTP within 
the GTHA. 

Our audit also included a review of the relevant 
audit reports issued by Metrolinx's internal audit 
department and the province's internal audit div- 
ision, which were helpful in determining the scope 
and extent of our audit work. 



Summary 




In the Greater Toronto and Hamilton Area (GTHA), 
congested roads and highways and public transit 
systems that are increasingly unable to meet the 
transportation needs of a growing population 
support Metrolinx's mandate of expanding and 
improving regional transportation across the 
area. We noted that other jurisdictions facing this 
problem have used similar stand-alone agencies 
to co-ordinate regional transportation planning. 
Accordingly, creating a co-ordinating agency like 
Metrolinx is a reasonable strategy toward imple- 
menting an effective, integrated and sustainable 
transportation network in a large urban centre. 

To successfully deliver on its challenging man- 
date, Metrolinx must ensure that: 

• individual projects under a regional trans- 
portation plan deliver transportation that is 



"seamless, coordinated, efficient, equitable 
and user-centred"; 

• a credible analysis of costs and benefits, 
based on objective and sound data, exists for 
each project; 

• the project management process ensures that 
risks are managed and that projects are deliv- 
ered cost-effectively and on time; 

• key stakeholders are sufficiently consulted 
with; and 

• clear targets are in place for achieving the 
transportation plan, and there is regu- 
lar reporting on progress in relation to 
these targets. 

Our review of the more significant projects in 
the early stages of the Regional Transportation Plan 
(RTP) identified a number of issues that must be 
addressed by Metrolinx, if it is to follow the best 
practices outlined above. Specifically: 

• We believe that Metrolinx's initial assump- 
tions about projected annual ridership on 
the Air Rail Link (ARL) may well be overly 
optimistic. Although a final decision has not 
been made on whether the ARL must recover 
its annual operating costs and any of its cap- 
ital construction costs, if operating the ARL 
on a break-even basis is indeed the objective, 
achieving that objective may not be feasible. 
Ministry of Transportation documentation 
indicated that a private-sector group that 
previously was the successful proponent for 
designing, building and operating the ARL 
was unable to secure financing for the venture 
because prospective lenders felt that despite 
all reasonable efforts to attract riders, the 
service might not generate enough revenues 
to be a viable business. A market assess- 
ment conducted by Metrolinx also suggests 
that ARL ridership may not meet the initial 
assumptions about ridership growth. 

• A region-wide integrated transit fare system 
is one of the RTP's key strategies. The Presto 
fare card now sponsored by Metrolinx is 
regarded as a key component in implementing 
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this strategy. Metrolinx's view is that the 
Presto fare-card system creates the underlying 
technology platform needed for fare integra- 
tion. But to date the card has not facilitated 
fare integration within GTHA transit systems 
because the fares across these systems are 
themselves not integrated. We noted the fol- 
lowing additional issues with respect to the 
Presto fare-card system: 

• When the Presto fare card was initially 
developed, the Toronto Transit Commission 
(TTC), which has over 80% of the GTHA's 
transit ridership, had not agreed to imple- 
ment Presto on its system. Along with the 
city of Ottawa, the TTC has now condition- 
ally approved Presto's adoption subject to 
satisfactory resolution of some key issues. 
However, to meet the requirements of 
Ottawa and Toronto, Presto Next Genera- 
tion (PNG) is currently being developed at 
an anticipated cost of $498 million. In total, 
more than $700 million could be paid to 
the contractor for developing the original 
Presto system and PNG. We acknowledge 
that Presto is intended to be the primary 
fare collection system on GO Transit and 
municipal transit agencies in the GTHA and 
therefore must be flexible enough to meet 
the needs of agencies and to adjust to new 
technologies as they become available; 
however, it will be among the more expen- 
sive fare-card systems in the world. 

• Rather than competitively tendering the 
development of PNG, Metrolinx decided 
to develop it by way of open-ended change 
orders under the existing vendor's contract. 
We believe that tendering would, at the 
very least, have informed Metrolinx of 
potential new developers and whether 
other vendors might have had more cost- 
effective technology solutions. 

• Since going into service approximately two 
years ago, Presto's overall usage within 



participating GTHA transit systems as 
of March 31, 2012, was only about 18%. 
Although seven of the eight municipal 
transit agencies in the 905 area code have 
implemented Presto, overall Presto usage 
on those systems was even lower, at only 
6%. These transit agencies currently cannot 
completely eliminate their old fare systems 
in favour of Presto because of some of the 
fare card's limitations. 
• The contract for the Presto base system 
contains 22 measures designed to gauge 
the contractor's performance in such 
areas as system availability and customer 
management. In 2011, the contractor 
failed to meet the set standard in nearly 
a third of the measures, but Metrolinx 
did not seek any of the related penalties 
stipulated in the contract. The contract 
also contains reliability measures for the 
devices used by the Presto base system, but 
neither the contractor nor Metrolinx tracks 
this information. 
• The two major projects related to the revital- 
ization of Union Station have experienced 
significant cost increases over their initial cost 
estimates. For instance, the cost of restoring 
the train shed could now reach $270 mil- 
lion — 25% over Metrolinx's initial estimate. 
Similarly, the cost of replacing the switches in 
the Union Station Rail Corridor could be more 
than twice the amount of the original pur- 
chase order, which totalled about $38 million. 
Although those GTHA municipalities and transit 
agencies we talked to questioned the priority given 
to some of the RTP's projects, they generally sup- 
ported the plan as currently conceived. However, 
some GTHA municipalities indicated that Metrolinx 
needs to provide more regular updates on the major 
projects under the RTP and on the RTP's overall 
status, including the strategies being considered to 
fund projects contemplated under the plan that are 
not yet funded. 
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Metrolinx welcomes the Auditor General's 
observations and recommendations. We have 
already taken action to address many of the 
Auditor General's recommendations, and 
we will continue our efforts to improve on 
our processes. 

The audit acknowledges that there is a press- 
ing need to deal with congestion and improve 
our transportation system. Metrolinx appreci- 
ates the Auditor General's observation that 
creating Metrolinx was a reasonable strategy 
toward implementing an effective, integrated 
and sustainable transportation network. 

The Greater Toronto and Hamilton Area 
(GTHA) is Canada's largest and fastest-growing 
urban region. With the GTHA generating 25% 
of Canada's GDP, the productivity impacts of 
congestion are significant. Today, GTHA com- 
mute times are among North America's longest. 
With population in the area growing by 100,000 
people per year, the GTHA is at risk of seeing 
commute times continue to increase. 

Metrolinx was created less than six years ago 
with a mandate to transform the way the region 
moves and a mission to champion and deliver 
solutions for the GTHA. Our Regional Trans- 
portation Plan ("The Big Move") is intended 
to lead integrated region-wide transit and 
transportation planning. 

We are committed to bringing forward 
new solutions to boost economic growth and 
help people and businesses move more easily 
throughout the region. This transformation has 
begun through a number of projects, such as the 
Mississauga bus rapid transit/403 transitway, 
the Toronto-York Spadina subway extension, 
York "VTVA rapidways, and Toronto light-rail 
transit projects, as well as improvements on GO 
Transit's Kitchener line and the construction of 
the Air Rail Link between Pearson International 
Airport and Union Station. 



With nine transit systems in the GTHA, all 
with various payment methods, Presto intro- 
duced a new fare-payment system that gives 
riders the convenience of being able to travel 
within the GTHA seamlessly and conveniently. 
As a regional fare card serving many transit 
providers, Presto is a unique product and one 
of the world's most advanced fare-card systems 
(similar to London's Oyster card, The Nether- 
lands' OV-chipkaart and the Chicago Card) . 
Today, throughout the GTHA, Presto is deployed 
among eight transit agencies and on GO Transit 
systems. As of October 2012, more than 350,000 
people were using the Presto card to travel 
throughout the multiple transit systems, and we 
have added an average of 22,000 customers per 
month over the last six months. When Presto is 
fully deployed on the TTC, its usage will grow to 
2.5 million customers. 



Detailed Audit Observations 



The Greater Toronto and Hamilton Area (GTHA) is 
Ontario's most populous region, with over 6 million 
people — a total that is expected to grow to 8.6 mil- 
lion by 2031. Currently, regional transportation 
within the GTHA primarily is served by several 
major expressways and by 10 different public tran- 
sit agencies, each with its own separate fare and 
infrastructure systems. 

Symptoms such as congested roads and highways 
and public transit systems that are increasingly chal- 
lenged to meet the needs of ever-growing population 
levels suggest that there is undoubtedly a pressing 
need within the GTHA to expand and improve the 
transportation system across the region, because 
the existing system may no longer be meeting 
the needs of the area's residents and businesses. 
Creating a co-ordinating agency like Metrolinx is 
a reasonable first step toward implementing an 
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effective, integrated and sustainable transportation 
network within the GTHA. Our research indicated 
that other major urban centres around the world 
have used similar agencies to co-ordinate regional 
transportation planning. 

There are a number of best practices that such 
agencies must follow to ensure the successful imple- 
mentation of effective transportation within their 
jurisdiction. Some of the key principles contained 
in Metrolinx's Regional Transportation Plan (RTP) 
that guide the delivery of the individual projects 
within the plan and the delivery of the overall plan 
itself include: 

• the individual projects should deliver trans- 
portation that is "seamless, coordinated, 
efficient, equitable and user-centred"; 

• the projects should be subject to a fair, clear 
and rigorous benefits case analysis process 
that considers financial, economic, environ- 
mental and social needs and impacts to ensure 
that the most optimal investment decisions 
are made; 

• the project delivery process should ensure 
that risks are managed and that projects are 
delivered cost-effectively and on time; 

• there should be sufficient consultation with 
key stakeholders; and 

• there should be clear targets for achieving the 
RTP and regular reporting on progress in rela- 
tion to these targets. 

In reviewing several of the major priority transit 
projects contemplated within the RTP's first 15 
years and in discussion with GTHA municipalities 
and transit agencies, we noted that Metrolinx has 
encountered challenges in successfully imple- 
menting some of these practices. The following are 
our specific observations. 

AIR RAIL LINK 

One of the more significant RTP projects currently 
under construction is the Air Rail Link (ARL) . As 
Figure 2 shows, the completed ARL will provide rail 
service between Canada's two busiest transporta- 



tion hubs: Union Station in downtown Toronto and 
Toronto Pearson International Airport. The ARL's 
target completion date is spring 2015, in time for 
the Pan/Parapan American Games to be held in 
Toronto in summer 2015. 

The 25 km line will primarily use GO Tran- 
sit's existing Georgetown South rail corridor. A 
new 3.3 km branch line ("spur") connecting the 
Georgetown South line with the airport is also 
being constructed. The ARL will have four stops: 
Union Station, the Bloor GO station, the Weston 
GO station, and Terminal 1 at the airport. Trains 
are expected to run every 15 minutes, seven days 
a week. A one-way trip is expected to take 25 min- 
utes. Metrolinx expects the ARL to be a premium 
rail service: some of the features being considered 
include on-board refreshments, Wi-Fi, power 
outlets for laptops, screens with flight information, 
self-service airline check-in machines and luggage 
facilities. Metrolinx has not yet determined the fare 
range for this service. (Fares will probably vary 
according to how far along the ARL a rider travels — 
that is, one, two, or three stops — but the specific 
fares have not yet been determined.) 

As Figure 3 shows, the ARL's estimated cost 
is about $456 million. A significant number of 
enhancements are also under way on the George- 
town South rail corridor, primarily to support an 
increase in the level of service for GO Transit on 
that line; but the ARL will also benefit from these 
enhancements. The estimated total cost of the 
enhancements on the Georgetown South rail cor- 
ridor is about $1.5 billion. 

Cost Recovery 

At the time of our audit, the province had not 
specifically required that Metrolinx recover the 
cost of operating the ARL from revenues that the 
service generates. The Ministry of Transportation 
(Ministry) informed us that Metrolinx would set the 
ARL's fare in consultation with the province. 

If operating the ARL on a break-even basis 
is indeed the objective, this may prove to be 
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Figure 2: Map of the Air Rail Link 

Source of data: Metrolinx 




Figure 3: Estimated Cost of the Air Rail Link ($ million) 

Source of data: Metrolinx 





^^^^^^ 


Description 




"Spur" and station in Terminal 1 at 
Pearson Airport 


168 


Trains 


98 


Other (stations, tracks and signals, etc.) 


190 


Total 


456 



a challenge for Metrolinx. In 2003, Transport 
Canada announced a private-sector group as 
the successful Public-Private Partnership (P3) 
proponent that would design, build and operate 
the ARL. However, the group was unable to secure 



financing for the venture because its lenders did 
not feel that they had sufficient protection from 
"no market" risk (that is, from a situation where, 
despite all reasonable efforts to attract riders, the 
service does not generate enough revenues to be a 
viable business). They perceived this project to be 
riskier than other infrastructure projects because 
there was no "pre-existing demonstrated revenue 
stream." The group proposed that the province 
assume the lenders' risk by purchasing ARL assets 
if the "no market" scenario arose. The province 
rejected this proposal, so the group walked away 
from the project. In 2010, the government decided 
that the province, through Metrolinx, would build 
and operate the ARL itself. 
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Metrolinx's preliminary estimate of the ARL's 
annual operating cost is approximately $30 million. 
However, according to Metrolinx, the cost could 
well be higher, because the service's exact nature 
has not been finalized, so some relevant costs may 
not have been identified yet. For example, the 
estimate does not include the annual access fee of 
approximately $5 million that GO Transit was going 
to charge the private-sector group for using the 
GO-owned Georgetown South rail corridor. As well, 
if the fare was to recover the capital cost of the pro- 
ject over time, we estimate this would approximate 
$20 million annually over a period of 20 years. If 
that amount is included as part of the ARL's operat- 
ing cost, the total cost to be recovered from fares 
each year would rise to about $50 million. 

Metrolinx's projection of annual ridership for 
the nine-month period of April 2015 to December 
2015 is 1.35 million (based on the assumption 
that the one-way fare for riding the full distance 
would be $20), and its estimate for the full first 
year is 1.8 million riders. The agency expects that 
ridership will increase by more than 65% to nearly 
3 million by year 3, capturing 10.3% of the surface 
access market — primarily travellers using taxis 
or those travelling by car who either park at the 
airport or are dropped off and/or picked up. If the 
aim was for the ARL to break even in its first year (a 
goal that has not yet been decided on), Metrolinx 
would have to charge about $28 for the full dis- 
tance (based on current ridership projections and 
estimated annual operating costs, including capital 
amortization). Under the private-sector group's 
proposal, the fare for a full one-way ARL trip was 
expected to be $27. If ARL ridership increases as 
projected by Metrolinx, the break-even fares over 
the longer term would be lower. But the following 
factors lead us to question whether ridership will 
actually grow as currently projected: 

• Although the projected capture rate of 10.3% 
is comparable to that of other North American 
airport rail services, these services differ signifi- 
cantly from the premium downtown-to-airport 
rail service that Metrolinx anticipates offering. 



Their one-way ticket prices range from only 
$1.60 to $13.00, compared to a ticket price 
for the ARL that may well cost $20 to $30. We 
believe that the ARL's high fare will negatively 
affect the projected ridership capture rate. 
• The results of a market assessment of GTHA 
residents conducted in November 2011 by 
Metrolinx revealed the following: 

• More than 90% of GTHA residents leave 
from and return to their home when travel- 
ling, so the added cost and inconvenience 
of getting to and from one of the three ARL 
stations with their luggage would prob- 
ably discourage some residents from using 
the ARL. 

• The ARL's likely price point may also be a 
concern. Although nearly 70% of potential 
riders currently using Union Station as an 
airport access or egress point indicated that 
they would probably use the ARL, nearly 
75% of those respondents who were GTHA 
residents also indicated that they would 
not be willing to take the ARL at a cost of 
$22.50 or more. As well, 60% of visitors 
and 90% of airport employees would not 
use the ARL at a cost of $22.50 or more. As 
would be expected, the percentages who 
would not use the ARL increased as the 
proposed price increased. 

Metrolinx advised us that it did take these fac- 
tors into consideration but still concluded that its 
ridership projections at these premium fare levels 
would be achieved. 



RECOMMENDATION 1 



Metrolinx should work with the Ministry of 
Transportation to clearly define the business 
model under which the Air Rail Link (ARL) 
should operate to ensure that the ARL will be 
a viable and sustainable operation. Given the 
importance of having a reliable estimate of 
projected ridership at the various possible fare 
levels, Metrolinx should periodically update its 
ridership forecast. 
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The Air Rail Link (ARL) will provide direct, 
reliable express service connecting Canada's 
two busiest transportation hubs: Union Sta- 
tion and Pearson International Airport. It is a 
priority project of the Regional Transportation 
Plan ("The Big Move") and is scheduled to open 
in 2015. 

Metrolinx agrees with the Auditor Gen- 
eral on the importance of reliable ridership 
forecasts, and independent analysis has been 
obtained to create ridership projections. 

As the ARL launch approaches, a number of 
significant decisions need to be made. Metrolinx 
will continue to use best-in-class ridership infor- 
mation to guide our internal decision-making 
and to inform our business model, and we will 
continue working with the Ministry of Transpor- 
tation to finalize the business model. 

As we would with any new service, Metro- 
linx will closely monitor the ARL over its first 
years of operation and make adjustments based 
on customer feedback we receive. 



The "Spur" Line 

As noted earlier, the ARL requires the construction 
of a new 3.3 km branch line, commonly referred 
to as the "spur," off of GO Transit's Georgetown 
South rail corridor connecting to a new passenger 
station in Pearson International Airport's Terminal 
1. In July 2010, when Metrolinx became responsible 
for ARL development, the government directed it 
to evaluate options for the delivery of the "spur" 
line and any related station work, including pos- 
sibly using the Public-Private Partnership (P3) 
model — which in Ontario is called the Alternative 
Financing and Procurement (AFP) model. Gener- 
ally, AFPs are contractual agreements between the 
government and the private sector under which 
the private-sector businesses provide assets and 
deliver services, and the various partners share the 



responsibilities and business risks. A Crown Agency, 
Infrastructure Ontario (I/O), oversees the delivery 
of all AFP projects in the province. 

Before deciding on the delivery model for a par- 
ticular project, I/O assesses which delivery model 
will provide the most value for money (VFM) . This 
VFM assessment compares the total project costs of 
two different delivery models (that is, AFP versus a 
traditional delivery method) . Four basic categories 
of cost make up the total project costs under each 
delivery model: base project costs (for example, 
construction costs), financing costs, the monetary 
value of the risks that will be retained under each 
delivery model, and any ancillary costs (such as 
legal, project management or engineering advisory 
fees) . Any positive difference between the AFP and 
the traditional delivery model represents the esti- 
mated monetary benefit from using the AFP. 

On the basis of a positive VFM assessment, I/O 
decided to use the AFP model in the delivery of the 
"spur"; a $128.6 million contract was subsequently 
awarded to a private-sector consortium. With 
respect to the procurement of the AFP contractor, 
we found that the process was competitive and fair 
to all respondents. 

In evaluating the VFM of procuring assets either 
in the traditional manner or by way of the AFP 
model, it is often the value of the risks retained 
under each delivery model that tends to tip the 
scale in favour of the AFP model. The VFM assess- 
ment concluded that using the AFP model for 
delivery of the "spur" would result in a net savings 
of about $20 million. While the total of the base 
project costs and ancillary costs under the AFP 
approach was estimated to be about $22 million 
higher, this was offset by an estimated $42 million 
in savings related to the transfer of risks under 
the AFP model. As Figure 4 shows, the two largest 
risks retained under the traditional delivery model 
are construction risk (the cost associated with 
construction delays) and design and tender risk 
(the cost incurred because of omissions in the 
original design and changes that are required after 
construction has started) . These two risks account 
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Figure 4: Valuation of the Retained Risks 

Source of data: Infrastructure Ontario 





Traditional Delivery 1 


AFP Delivery 2 


1 Retained Risks 


($000) 


(%) 


($ 000) 


(%) 


Construction 


21,160 


42 


1,327 


15 


Design and tender 


12,618 


25 


1,525 


18 


Policy/strategy 


6,518 


13 


1,688 


20 


Site conditions/environmental 


5,428 


10 


1,107 


12 


Project agreement 


871 


2 


581 


7 


Permits and approvals 


354 


1 


64 


1 


Other 


3,722 


7 


2,310 


27 


Total 


50,671 


100 


8,602 


100 



1. Under the traditional delivery model, the province bears all the risks. 

2. Under the AFP delivery model, risks are shared between the province and the contractor. 



for two-thirds of total risk retained under the trad- 
itional delivery model. 

Of concern to us is the process used to assign 
values to the various risks seen as being retained 
under the two delivery models. Specifically: 
• The values assigned to the risks seen as 
retained under both delivery models were 
derived based on the judgment of I/O staff, 
Metrolinx staff and a consulting firm that 
devised the probabilities and impacts associ- 
ated with the various risks. While we acknow- 
ledge that I/O has significant experience in 
capital projects such as hospitals, courthouses 
and other buildings, we saw no evidence that 
the estimates of the risks of delivering the 
"spur" under traditional procurement were 
based on actual experience of similar, trad- 
itionally procured transportation projects. For 
instance, over the past eight years GO Transit 
has completed a number of large and complex 
rail and grade-separation projects. The actual 
experience from these could have been used 
to assess the reasonableness of the values 
assigned to the risks that are seen as being 
retained under the traditional delivery model, 
especially given the significant $42 million 
risk differential between the two procurement 



alternatives, which was the deciding factor in 
going with the AFP approach. 

• Because Metrolinx would be locked in very 
early on the specifications of the project 
under the AFP model, the additional cost that 
could be incurred as planning and design 
progress because of subsequent changes iden- 
tified and considered necessary could also 
have been considered in the allocation and 
valuation of the risk retained under the AFP 
delivery model. 

• I/O's procedures allowed the consulting firm 
that devised the project's risk allocation matrix 
to later bid on a contract to provide engineer- 
ing and technical advisory services to support 
the planning and procurement of the "spur" 
line under the AFP delivery model. The con- 
tract was subsequently awarded to this firm. 



RECOMMENDATION 2 



When assigning values to transferable risks in 
the evaluation of value for money between pro- 
curing assets by way of the traditional method 
or by way of the Alternative Financing and Pro- 
curement (AFP) model, actual experience from 
recent traditional infrastructure procurements 
and AFPs should be thoroughly assessed. 
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I INFRASTRUCTURE 
| ONTARIO RESPONSE 

The process for developing the value-for-money 
analysis was robust and based on current best 
practices. An experienced transportation con- 
sulting firm created a standard transportation 
risk matrix based on the firm's analysis of indus- 
try data and on its own in-house experience. 
The matrix values were then further reviewed 
and revised by Metrolinx and Infrastructure 
Ontario in consultation with the external advis- 
ers. Given the design and construction risks and 
the scheduling risk for this project, the Alterna- 
tive Financing and Procurement (AFP) model 
was determined to deliver value for money 
compared to traditional delivery. 

Infrastructure Ontario ensured that strict 
controls were in place to maintain objectivity 
of the firm conducting the value-for-money 
analysis and the engineering advisory services. 
For future projects contemplated under the 
Regional Transportation Plan, Infrastructure 
Ontario and Metrolinx will continue to use risk 
workshops to fully assess the actual experience 
of transportation projects procured under trad- 
itional methods, including new developments in 
procuring large transportation projects both in 
and outside of Canada. 



PRESTO FARE SYSTEM 

Currently, there are 10 public transit agencies in 
the GTHA, each with its own fare structure and a 
separate system for collecting fares. As a result, for 
example, a person travelling from a local bus in 
one GTHA municipality to the GO Train and then 
to the City of Toronto transit system must pay three 
different fares. One of the RTP's key strategies is to 
"implement a region-wide integrated transit fare 
system by 2012 that allows users to pay a seam- 
less, integrated fare for all transit systems across 
the region." 



In 2002, the Ontario Ministry of Transporta- 
tion, in conjunction with GO Transit and the GTHA 
municipalities, began researching the development 
of a regional fare card, now called Presto. Presto, 
which is now one of Metrolinx's priority transit 
projects, allows transit riders to load amounts 
onto a reloadable plastic card (the size of a credit 
card) and pay their fares by tapping the card on 
electronic card readers. Amounts ranging from $10 
to $1,000 can be loaded onto the card online or in 
person at participating customer service outlets. 

A number of guiding principles for Presto's 
development and implementation were identified 
in 2002, including the following: 

• The fare system should, where possible, use 
off-the-shelf products whose components can 
be purchased from multiple sources. 

• The fare system needs to have the ability 
to add new transit participants of any size 
without major modifications to its core 
operational structure. 

Project Cost 

In October 2006, the Ministry of Transportation 
signed a 10-year, $250-million contract with a 
vendor to design, develop and operate Presto for 
the GTHA. The $250 million is composed of about 
$150 million in capital development costs, $82 mil- 
lion in operating costs, and taxes of about $20 mil- 
lion. The province anticipated that all GTHA transit 
systems, including GO Transit, would use this card. 
To encourage this, the province has indicated that 
the transfer of gas-tax funding to municipalities 
would be contingent on their adopting and staying 
with Presto. 

However, when the agreement was signed the 
Toronto Transit Commission (TTC), which has over 
80% of the GTHA's transit ridership, had not agreed 
to implement Presto on its system. Anticipating 
that the TTC would eventually opt in, the agree- 
ment with the vendor stipulated that the original 
base system would be built with the capability to 
expand to meet the needs of all Ontario transit 
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providers without the need for significant modifica- 
tions to core systems. The agreement also stated 
that full rollout of the Presto base system was to be 
completed by October 2010. Although GO Transit 
started adopting Presto on its system in Novem- 
ber 2009, followed by the GTHA municipalities 
in the 905 area code in May 2010, full rollout 
(meaning that Presto base was implemented on 
all intended transit systems, and the fare system 
was functioning as planned) did not occur until 
February 2012. 

In 2007, the City of Ottawa approved the 
implementation of the fare card on its transit 
system based on an agreed functionality to be 
provided by Presto. In November 2009, the TTC 
also conditionally approved the adoption of the fare 
card subject to the satisfactory resolution of some 
key issues (such as the system meeting the TTC's 
business needs and being affordable from both 
a capital and an operating perspective) . Rather 
than expanding the Presto base system to meet the 
requirements of Ottawa and Toronto, as had origin- 
ally been planned, the Ministry of Transportation 
decided to develop a new system, Presto Next 
Generation (PNG). 

The Presto base system, contrary to the guid- 
ing principles established for its development, 
works on a closed proprietary model: that is, the 
contractor provides and controls the central system 
and other infrastructure for the fare-card operation. 
Changes can be made only through change orders 



after direct negotiation with the contractor. The 
Ministry and Metrolinx believe that developing 
PNG on an open architecture framework, as had 
been originally intended in 2002, will allow for 
more procurement options should there be a need 
to add additional functionalities in the future. At 
the time of our audit, PNG was initially expected to 
be rolled out in Ottawa in July 2012, but the rollout 
was postponed to February 2013. Metrolinx was 
also negotiating with the TTC to become a fully par- 
ticipating PNG member in time for the 2015 Pan/ 
Parapan American games. 

As Figure 5 shows, PNG's anticipated cost 
includes $498 million specifically for the system's 
development and $152 million to be paid to the 
vendor for operating the system and running a 
call centre until 2016, for an anticipated total of 
$650 million. The total cost of developing Presto 
base and PNG could well reach $700 million. As of 
March 31, 2012, about $360 million of this amount 
had been spent on system development costs, includ- 
ing about $40 million in internal charges incurred 
by the Presto office at Metrolinx, which has a staff 
of about 60. Additionally, although one of the key 
reasons for the new PNG is to meet the TTC's needs, 
at the time of our audit the TTC had not yet formally 
signed on to using the fare card. The TTC indicated 
to us that one of its main reasons for not yet signing 
on was that Metrolinx and the TTC had not yet final- 
ized the TTC's service-level requirements and how 
the service levels will be achieved through PNG. 



Figure 5: Estimated Presto Project Costs ($ million) 

Source of data: Metrolinx 





Capital 


Operating 




Presto Base 


PNG 


Total 


Presto Base 


PNG 


Total 


Original contract (2006) 


149.0 




149.0 


82.5 




82.5 


Nine separate requests to Metrolinx Board 
(August 2009-February 2012) 


54.0 


154.0 


208.0 


19.0 




19.0 


Additional request (April 2012)* 




344.0 


344.0 




152.0 


152.0 


Total 


203.0 


498.0 


701.0 


101.5 


152.0 


253.5 



* At the time of our audit, the Board had approved only $48.5 million of this additional amount and had asked Metrolinx management to carry out further due 
diligence on whether value for money is being received with respect to this expenditure. 
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RECOMMENDATION 3 



Metrolinx should ensure that it formally consid- 
ers the risks of continuing with the development 
of Presto Next Generation (PNG), given that the 
specific business requirements of the Toronto 
Transit Commission (TTC) for using PNG on its 
transit system and the costs for which the TTC 
would be responsible have not yet been formally 
agreed to. 

| METROLINX RESPONSI 

Metrolinx has been working with the TTC since 
2009 to define the business requirements of 
Presto. The TTC approved the implementation 
of Presto on November 23, 2011, and authorized 
the execution of all necessary agreements. This 
was reaffirmed on May 1, 2012, and we antici- 
pate completing a master agreement with the 
TTC in 2012. The discussions regarding the mas- 
ter agreement have included both the operating 
requirements and financial arrangements; thus, 
these discussions have informed the develop- 
ment of Presto Next Generation. As well, Presto 
has been installed at 14 of the TTC's highest- 
volume subway stations and it is used daily by 
more than 8,000 riders. It should be also noted 
that the TTC has estimated that when Presto is 
fully operational, costs for fare collection could 
be reduced by up to $10 million annually from 
current levels. 



Fare Integration 

As noted earlier, a key success factor of the RTP was 
to develop a seamless and integrated fare for all 
transit systems across the GTHA that would allow 
riders to cross regional and municipal boundaries 
using different transit systems by paying just one 
fare rather than having to pay an individual fare for 
every system travelled on. The Presto base system, 
apart from facilitating fare arrangements between 
GO Transit and bus systems in municipalities within 



the 905 area code, has not in itself facilitated the 
integration of fares across GTHA transit systems. 
Currently, it is being used only as an "e-purse" 
that allows users to tap the card to the Presto card 
reader and automatically be billed the individual 
fares of the participating GTHA transit systems. 

GTHA municipalities and transit systems indi- 
cated to us that as long as transit funding remains 
a municipal responsibility, fare integration will be 
difficult to achieve, because GTHA municipalities 
are not willing to absorb the cost of the subsidies 
that an integrated fare system may entail. For 
example, the fare arrangement between GO Transit 
and transit systems in the 905 area code costs GO 
Transit approximately $7 million annually, because 
GO Transit riders pay a reduced local transit fare to 
encourage these riders to use local transit instead 
of cars to arrive at their respective GO stations, 
with GO Transit paying the difference to the 
respective municipalities. 

Presto Usage 

As Figure 6 shows, at the time of our audit, the 
Presto card was accepted within the GTHA as 
follows: on seven of the eight municipal transit 
systems within the 905 area code, as well as on 
GO Transit (both rail and bus) and at 14 of the 69 
TTC subway stations (but not on any TTC buses or 
streetcars) . As of March 31, 2012, despite the sub- 
stantial investment in the Presto base system and 
despite Metrolinx being six years into a 10-year con- 
tract for the system's development and operation, 
Presto's overall usage within participating GTHA 
transit systems was only about 18%. 

Overall Presto usage was even lower — only 
6% — within the participating GTHA transit agen- 
cies in the 905 area code. Several of these agencies 
indicated to us that a good portion of their rider- 
ship (nearly a third in some regions and municipal- 
ities) are considered low-income and either cannot 
afford to load the minimum $10 currently required 
by the Presto card or do not have bank accounts or 
credit cards and therefore cannot load the cards 
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Figure 6: Presto Usage Rate for March 2012 

Source of data: Metrolinx and GTHA transit systems 
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Presto 
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In-service Date 


Ridership 


Ridership 


Ridership (%) 


GO Transit System 


Rail 


Aug. 8, 2011 


4,169,337 


1,788,037 


43 


Bus 


Sept. 12, 2011 


1,506,716 


242,335 


16 


GO Transit System Subtotal 




5,676,053 


2,030,372 


36 


TTC (at select subway stations) 


Jan. 1,2011 




252,025 




905 Transit Systems 


Mississauga 


Apr. 4, 2011 


3,315,817 


140,655 


4 


Brampton 


Apr. 4, 2011 


1,593,637 


231,770 


15 


York* 


Apr. 4, 2011 


1,755,264 


64,843 


4 


Hamilton 


Apr. 4, 2011 


1,969,218 


67,258 


3 


Durham 


Apr. 4, 2011 


964,168 


31,669 


3 


Burlington 


May 10, 2010 


204,729 


35,179 


17 


Oakville 


May 10, 2010 


258,310 


41,614 


16 


905 Transit Systems Subtotal 




10,061,143 


612,988 


6 






15,737,196 


2,895,385 





* Because of a transit strike and its effect on ridership for March 2012, we used April 2012 data for York Region. 



online. The initial $6 charge for the card also acts as 
a disincentive for these riders to migrate to Presto. 
For these reasons, GTHA transit systems within the 
905 area code may need to maintain some form of 
disposable fare media (defined as media that are 
good for only a short term — either a single trip or 
multiple trips over the course of a day — such as 
tickets, tokens or day passes) . These agencies cited 
the following additional reasons why they cannot 
completely eliminate their existing fare systems and 
force their ridership to migrate to Presto: 
• Not all Presto-related transactions can be 
done online. For instance, to load monthly 
passes or if a student or senior wants to 
register for a card, the rider must physically 
go to a Presto location. But many municipal 
transit systems have only one location where 
in-person Presto transactions can be carried 
out. GTHA municipalities and transit systems 
within the 905 area code indicated to us that 
point-of-sale terminals installed in such loca- 
tions as convenience stores would provide 
riders with greater access, but given the cur- 



rent low demand and the nearly $5,000 cost 
of installing a single terminal, very few are 
being installed. 

• One region contracts out routes used by about 
30% of its ridership to the TTC, but the TTC 
currently does not accept the Presto card on 
its buses. 

• GTHA municipalities within the 905 area code 
that have a significant population of university 
students and offer students a special transit 
pass for the university term under their own 
fare system cannot currently do so on Presto. 

These transit agencies also raised concerns 
about the quality of the Presto equipment installed 
on their vehicles and the repair costs for what are 
deemed "out-of-warranty" damages. Because the 
equipment is proprietary, if "out-of-warranty" 
repairs are needed, municipal transit agencies 
can turn to only one approved supplier under the 
existing contract. The contract does not provide 
specific pricing for the different types of repairs. 
In our discussions, municipal transit agencies 
cited examples of quotes for repairs that they had 



Metrolinx— Regional Transportation Planning 



received from the contractor that they felt were 
unreasonably high. 

Lastly, the transit agencies indicated to us that 
the system currently lacks back-end support for 
reporting and financial reconciliation of trans- 
actions. To obtain ridership information, many 
transit agencies have had to design their own 
programs for extracting information from a data 
dump provided by Presto. The transit agencies 
also indicated that they have little assurance that 
the system is capturing all riders who use their 
respective systems. 



RECOMMENDATION 4 



To ensure that the Presto base system and 
the Presto Next Generation system meet the 
objective of facilitating a seamless, integrated 
fare for all transit systems across the GTHA, 
Metrolinx should: 

• work with the provincial government and 
GTHA municipalities to resolve the issue of 
subsidizing fare integration so that progress 
can be made on implementing an integrated 
fare system; and 

• work with GTHA municipalities and regions 
to resolve outstanding issues related to 

the operation of Presto that inhibit riders' 
use of the fare card within their respective 
transit systems. 



| METROLINX RESPONS 




The Presto system is a foundational step toward 
developing an integrated fare system across the 
GTHA. The system is currently being deployed 
across the GTHA and in Ottawa. A staged 
deployment provides the least amount of risk as 
the system is implemented. 

As the Presto system grows, Metrolinx will 
continue to work with the province and with the 
municipalities involved to develop strategies for 
increasing customer usage as well as to enhance 
the level of integration, up to and including 
the development of a common fare structure. 



For instance, Metrolinx is working closely with 
Brampton Transit to retire that municipality's 
current payment systems in 2013. In Durham, 
we are also working to convert student riders 
to Presto in 2013. Similar strategies are being 
developed for each municipality in the GTHA. 

On GO Transit, Presto usage will increase 
further in 2013 with the retirement of the GO 
monthly pass. 



Project Procurement 

As noted earlier, in October 2006 the Ministry of 
Transportation signed a 10-year, $250 million con- 
tract with a vendor to design, develop and operate 
the Presto base system. This contract was procured 
through a competitive process and subjected to a 
fairness review that concluded that the process 
was conducted in a procedurally fair, open and 
transparent manner. However, with respect to the 
development of the PNG system, Metrolinx was 
unable to provide evidence supporting its 2009 
decision to develop this system through change 
orders to the existing Presto contract rather than 
through a competitive tender. As noted in Figure 5, 
earlier, at the time of our audit, Metrolinx had 
Board approval to spend an additional $227 million 
($208 million capital plus $19 million operating); 
of this amount, $154 million was for PNG. In April 
2012, Metrolinx went to the Board for approval of 
an additional $496 million ($344 million capital 
plus $152 million operating) for PNG. 

After deciding to develop PNG using change 
orders, Metrolinx hired Ontario's former Integrity 
Commissioner to review the appropriateness of this 
decision. In September 2011, Metrolinx also hired 
a consulting firm to assess this additional invest- 
ment in PNG and to assess whether value for money 
(VFM) would be achieved. The commissioner's 
February 2012 letter concluded that there was no 
compelling reason to restart the procurement pro- 
cess on PNG if the results of the VFM review were 
positive. The VFM review compared the per capita 
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cost of fare-card systems around the world with the 
per capita cost of the Presto base system and PNG 
after considering the capital portion of the $227 mil- 
lion in the first set of change orders and concluded 
a positive VFM on the basis that Presto's per capita 
cost ranked in the middle. However, if the capital 
portion of the additional $496 million expected to 
be incurred had been included in the analysis, Presto 
base and PNG combined would turn out to be one of 
the more expensive fare-card systems in the world. 

In April 2012, citing concerns about the request 
for an additional $496 million in spending authority 
for PNG, the Board asked Metrolinx's manage- 
ment to carry out further due diligence on PNG. In 
response, Metrolinx asked the same consulting firm 
for a second VFM review of PNG. This second review 
concluded that, although reprocurement "may drive 
pricing benefits," it incurred a significant risk of 
not meeting the timelines for the development of a 
new fare card, because procuring, developing and 
implementing a new system would take more than 
24 to 48 months. Furthermore, having a new vendor 
build the system would introduce significant other 
project and operational risks that could hinder the 
efficient delivery of the entire system. 

We note with respect to these concerns about 
reprocurement that the existing plan for developing 
and implementing PNG, having begun in 2009, 
will take about four years anyway because rollout 
of the system in the City of Ottawa is not expected 
until February 2013. We also note that there may 
have been value in considering the possibility of 
alternative procurement options. In this regard, for 
example, the TTC had an agreement in principle 
with a company in 2011 to develop an open-fare 
payment system. Under the terms of the agreement, 
the TTC would not have had to pay any capital 
costs up front. Instead, the vendor was willing to 
take a percentage of the revenues collected by the 
open-fare system. In its proposal to the TTC, the 
vendor had also agreed to make its system compat- 
ible with the existing Presto base system. The TTC 
abandoned this option after the province confirmed 
that provincial gas-tax funding and provincial 



funding for the purchase of new streetcars and the 
rapid transit system on Eglinton Avenue would be 
contingent on the TTC signing on to Presto. 

As noted earlier, at the time of our audit, 
Metrolinx was unable to provide evidence that 
it had explored alternative procurement options 
at the time the decision was made to develop 
PNG. We questioned whether tendering the 
new system's development would have, at the 
very least, informed Metrolinx of the range of 
options and what a reasonable cost would be for 
developing PNG. 



RECOMMENDATION 5 



To ensure that Metrolinx complies with the 
intent of the government's policy of open, 
competitive procurement, all value-for-money 
considerations and an appropriate business-case 
justification should be completed and approved 
by Metrolinx's Board and the Ministry of Trans- 
portation before any decision on the procure- 
ment of significant transportation projects is 
finalized, especially if retendering the projects is 
not considered to be a viable option. 

METROLINX RESPONSE 

A comprehensive review of Presto Next Gen- 
eration (PNG) technology was undertaken by 
independent advisers, who confirmed that the 
development of PNG was fair and created value 
for money (VFM) . The VFM analysis identified 
concerns with retendering the work, including 
increased project costs and risks associated 
with the introduction of new vendors, increased 
project timelines and the loss of efficiency. As 
we move forward, we are reducing the role of 
the contractor, increasing the amount of work to 
be procured in separate competitive processes 
by about $200 million. As the technology 
continues to advance, more functionality and 
conveniences will be added for Presto custom- 
ers, including additional services and other 
payment methods. 
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Metrolinx complies with all provincial 
requirements for an open, competitive procure- 
ment process, and has ensured that VFM con- 
siderations as well as appropriate business-case 
justification are part of the decision-making pro- 
cess. However, Metrolinx does agree with the 
Auditor General's recommendation that VFM 
considerations and an appropriate business-case 
justification should be completed and approved 
before making any decision on a project's pro- 
curement strategy. Metrolinx has implemented 
this recommendation with the expansion of 
Presto to the TTC. 

With respect to Metrolinx's $700 million 
investment in Presto, approximately $275 mil- 
lion is expected to be recovered from the TTC, 
OC Transpo and the GTHA transit agencies in 
the 905 area code for assets specifically used in 
the provision of service to their customers. 



Change-order Management 

The $250 million contract for the original Presto 
base system is a 10-year fixed-price contract to 
deliver an electronic fare-card system for GO 
Transit and the seven participating GTHA muni- 
cipalities in the 905 area code. However, since 
the contract's execution in 2006, a total of 330 
change requests, adding $146 million to total costs, 
have been made under the contract. Of these, 281 
change orders totalling $45 million relate to fixes or 
enhancements to the Presto base system that were 
requested by either Metrolinx or the participating 
transit agencies, with the balance relating to PNG. 
For example, in one case, the contractor charged 
$7 million to make nine enhancements to the Presto 
base system and at the same time fix 40 defects 
that had been identified in the system's original 
development. It is reasonable for a contractor to 
charge for change orders that enhance or alter the 
system from its original agreed-upon design speci- 
fications, but the contractor should not be charging 
for change orders that correct identified defects 



in the system's original development. In the case 
of the above example, the documentation was not 
clear enough to determine whether the payment 
related to enhancements to the system or to the 
correction of defects, which should have been done 
at no cost. 

GTHA transit systems in the 905 area code 
that we met with indicated that changes to the 
Presto base system often seemed too costly and 
that change requests were not always completed 
on what they felt was a reasonably timely basis. 
The consulting firm mentioned previously that was 
hired to assess PNG noted in its December 2011 
report that Metrolinx's change-order process lacks 
detailed cost breakdowns and pricing methodolo- 
gies and that no formal budget estimates had been 
prepared for requested changes that could be used 
to assess the reasonableness of the amount being 
billed for each change order. 



RECOMMENDATION 6 



In order to effectively manage the cost of change 
orders related to the Presto base and Presto Next 
Generation systems, Metrolinx should: 

• implement a process that distinguishes 
between change orders that amend the 
systems from their original specifications in 
the contract and those that correct identified 
defects in the systems' original develop- 
ment, and allow the contractor to charge 
for only those change orders that pertain to 
requested changes or enhancements to the 
original design specifications; and 

• prepare internal cost estimates for each 
change order to enable the reasonableness of 
the amount charged by the contractor to be 
knowledgeably assessed. 

| METROLINX RESPONSE 

Metrolinx agrees with the Auditor General on 
the importance of effectively managing the cost 
of change orders. 
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A consulting firm recently completed a 
review of the change-order process. Although 
this process was found to be efficient and effect- 
ive, opportunities for improvement were identi- 
fied. Metrolinx has now implemented controls 
over the change-order process, including gener- 
ating internal cost estimates before completing 
negotiations on change orders and increasing 
documentation of each change order's purpose 
and scope. 

Metrolinx is in the process of implementing 
additional accountability measures, which will 
be independently reviewed to ensure that these 
additional measures have been implemented 
and to identify whether any further improve- 
ments are necessary. 

With regard to the $7 million charges noted 
by the Auditor General, Metrolinx has reviewed 
these charges and has confirmed that they were 
related to system enhancements rather than to 
the correction of defects. 



Other Presto Issues 

We noted several additional issues in our review 
of the development of the Presto base and 
PNG systems: 

• Ownership of certain key components of the 
Presto base and PNG systems is currently 
unclear. The contractor maintains that it owns 
the system and can therefore market it to other 
parties. Metrolinx has asked the contractor 
to pay $25 million for the right to market the 
Presto base and PNG systems anywhere in the 
world (including to non-government entities in 
Canada), while Metrolinx can market the sys- 
tems only to government entities in Canada. 
If the ownership and marketing rights of the 
system are not resolved, Metrolinx risks losing 
key components of the Presto base and PNG 
systems at contract termination, which would 
render the rest of the systems inoperable. 
Metrolinx advised us in mid-October 2012 that 



it was finalizing the negotiation of a letter of 
intent to secure ownership in Canada of intel- 
lectual property relating to these systems. 

• The contract for the Presto base system 
contains 22 measures designed to gauge the 
contractor's performance in such areas as 
system availability, customer management 
and the management of the Presto devices 
(for example, the card reader). In 2011, the 
contractor failed to meet the set standard 
in nearly a third of the measures. However, 
Metrolinx did not seek any of the remedies 
stipulated in the contract for these failures 
and indicated to us that it will just continue 
to monitor performance until system usage 
reaches maturity. The contract also contains 
reliability measures for the equipment 
used by the Presto base system, but neither 
the contractor nor Metrolinx tracks this 
information. Therefore, Metrolinx cannot 
determine whether the equipment is meet- 
ing the reliability measures, which is all the 
more important because some municipalities 
we talked to expressed concerns about 
equipment reliability. 

• During the period May 1, 2007, to April 30, 
2012, independent contractors were used 
in senior positions with signing authority 
to supervise other consultants. The amount 
paid to these contractors was $4.2 million. 
But the government's procurement directive 
specifies that "consultants must not perform 
functions normally assumed by management, 
including supervising and hiring staff and 
other consultants." At the time of our audit, 
Metrolinx was in the process of terminating 
these relationships. 



RECOMMENDATION 7 



To ensure that the Presto base and Presto Next 
Generation systems remain available for use 
after the end of the existing contract, Metrolinx 
needs to finalize its current negotiations with 
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the contractor to ensure that it secures owner- 
ship of these two systems. If the contractor fails 
to meet the performance standards stipulated in 
the contract, Metrolinx should have a valid justi- 
fication for not applying the available remedies 
and penalties set out in the contract. 

| METROLINX RESPONSI 

Metrolinx is in advanced negotiations with the 
contractor to safeguard ownership rights of the 
intellectual property created and expects these 
negotiations to be successfully concluded in 
October 2012. The expected agreement provides 
for the use of the current and future system 
in perpetuity. 

As the system has matured, Metrolinx has 
been engaged in an extensive internal review 
of contractor performance and has developed a 
plan that provides for a more rigorous monitor- 
ing of key service measures, as well as appropri- 
ate remedies and penalties for situations where 
these measures are not met. 



UNION STATION REVITALIZATION 

Union Station, federally designated as a National 
Historic Site and a Heritage Railway Station, is 
Canada's busiest rail transportation facility, serv- 
ing more than 250,000 passengers daily who use 
the services of the TTC, GO Transit, Via Rail and 
Amtrak. In August 2000, GO Transit and the City of 
Toronto bought the facility from Toronto Terminals 
Railway Company. Specifically, GO Transit bought 
the three-mile rail corridor leading in to the sta- 
tion, the platforms and the train shed, and Toronto 
bought the heritage building, including the GO 
Transit concourse area. 

Revitalizing Union Station is one of the priorities 
in the RTP's first 15 years: it consists of a series 
of projects. Two of the more significant projects 
include restoring the train shed and replacing 
switches in the Union Station Rail Corridor (USRC) . 



We reviewed these two projects and made the 
following observations. 

Train Shed Restoration 

Under the federal Heritage Railway Stations Protec- 
tion Act, Parks Canada must approve all rehabilita- 
tion work planned for Union Station, including its 
train shed (the structure that shelters the station's 
platforms and the tracks alongside them) . For 
instance, between 2005 and 2010, Parks Canada 
approved the replacement of approximately 20% 
of the centre portion of the train shed roof with 
a glass atrium. Approval was also received to 
replace the existing roof with an eco-friendly green 
roof on either side of the atrium. Although the 
initial estimate for the cost of restoring the train 
shed — including design, administration and con- 
struction — was $215 million, when the bids for the 
work came in, the lowest bids totalled $242 million. 
The majority of the difference was in the construc- 
tion work, for which the initial estimate had been 
$165 million, but the lowest bid was $196 million. 

The project was initially scheduled to be com- 
pleted in November 2014, but delays experienced 
during the design and construction phases could 
push completion to December 2016. We noted that 
the construction contract contained no incentives 
for on-time completion or liquidating damages in 
the case of delays. Metrolinx chose not to include 
such clauses because it felt that the contractor 
did not have full control of the site (that is, Union 
Station would be fully functioning during construc- 
tion, and no more than two tracks or platforms 
would be taken out of service at any time) . 

In the construction contract for the restora- 
tion of the train shed, Metrolinx retained, among 
other risks, the risk associated with concealed 
or unknown conditions that may arise during 
construction. It was felt that if such risks were 
transferred to the contractor, either contractors 
would not bid on the work or the premium charged 
to cover unforeseen risks would be cost-prohibitive. 
Metrolinx included a $15 million contingency in the 
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contract's original price (8% of its total value) for 
any unforeseen situations. Metrolinx then approved 
an increase in this contingency to $36 million (18% 
of the contract's total value) by not requiring the 
contractor to carry out work originally stipulated 
in the contract totalling $21 million and moving 
this amount to the contingency. As of May 2012, 
$30 million of this contingency had been allocated 
to the contractor, leaving approximately $6 million. 
In April 2011, Metrolinx had also received Board 
approval to increase the construction contract's 
price by an additional $30 million by way of change 
orders. As of August 2012, change orders totalling 
$18.1 million had been issued to the contractor. 
Most of this amount was requested by the con- 
tractor to cover any additional overheads caused by 
extending the project completion date by another 
two years. The cost of restoring the train shed 
could now reach $270 million — 25% more than 
Metrolinx's initial estimate. We note that nothing in 
the current agreement prevents the contractor from 
coming to Metrolinx for even more funds over the 
remaining term of the contract. 

Metrolinx informed us that as a hedge against 
unforeseen situations, construction contracts 
commonly provide for contingencies of 5% to 15% 
of the contract's original value. This contract's 
contingencies and the change orders totalled nearly 
$55 million, or 28% of the construction contract's 
original price — almost twice the high end of the 
norm. Significant price changes in contracts can 
occur because of poor planning, inadequate pro- 
cesses for estimating the initial cost projections, 



weak monitoring of the project or a combination 
of these problems. In 2011, the province's Internal 
Audit Division reviewed Metrolinx's budgeting 
and forecasting process and found that the capital 
budgeting and forecasting processes were not well 
established, and also that recent years' budget-to- 
actual results suggest that Metrolinx may need to 
re-evaluate how project costs and/or contingencies 
are determined. 

Switch Replacement Project 

Since June 2000, GO Transit (a division of Metro- 
linx) has had a single-sourced agreement with 
Toronto Terminals Railway Company (TTR) — 
which previously owned the Union Station Rail 
Corridor (USRC) — to conduct routine USRC 
maintenance and rail traffic control services for an 
annual fee. In June 2006, a new agreement with 
TTR for a further six years, at approximately $7 mil- 
lion annually, was entered into. Under this agree- 
ment, in 2006, GO Transit issued a purchase order 
totalling nearly $38 million to replace about 100 
switches within the USRC over a six-year period. 
According to GO Transit, the new switches allow 
trains to run faster into and out of Union Station, 
thereby providing additional train capacity and 
more efficient train operation. As Figure 7 shows, 
beyond the initial 2006 purchase order issued for 
switch replacement, three additional purchase 
orders totalling over $50 million were also issued 
(in 2008, 2010 and 2011, respectively). Metrolinx 
advised us that the significant cost increases 



Figure 7: Approved Purchase Orders for Switch Replacement Project ($ million) 

Source of data: Metrolinx 



1 P0# 


Date 


Description 


Amount 


1 


June 9, 2006 


Removal and installation of approximately 100 new switches 


37.6 


2 


Nov. 5, 2008 


Cost increases for changes in regulations and standards, staff training and material costs 


14.0 


3 


July 20,2010 


Cost increases for delays due to testing and need for redesign and staff costs 


15.0 


4 


Mar. 14, 2011 


Cost increases associated with the more complicated switches to be installed in 2011 and 
2012 


23.0 


Total 






89.6 
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resulted primarily from changes in regulations 
and safety standards, delays due to the switches 
being replaced because of their age, and the instal- 
lation of the more complicated switches later in 
the project. As of May 2012, TTR had installed 
approximately 90 switches at a total cost to date of 
$76.1 million — more than twice the total amount of 
the original purchase order. 

This project was managed by an external con- 
sulting firm under a contract that expired in 2010. 
Although the firm handed over information on the 
project to Metrolinx before ceasing to work on the 
project, the information was not well organized, 
making the search for details on this project very 
difficult and time-consuming. As a result, other 
than requests forwarded to Metrolinx's Board 
for funds to pay for cost increases related to the 
project, no other documentation was available to 
support the reasonableness of such significant cost 
increases. TTR also works on other projects within 
the USRC for Metrolinx, and we noted that the 
contractor's invoices did not always clearly specify 
which project the work relates to, making monitor- 
ing of project costs very difficult. 

In our 2007 report on GO Transit, we expressed 
concern over the fact that for work in the USRC, GO 
had not actively sought other qualified suppliers or 
considered the feasibility of developing in-house 
expertise to prevent becoming overly dependent on 
the USRCs previous owner. We continue to have 
this concern. 



RECOMMENDATION 8 



To ensure that projects under the Regional Trans- 
portation Plan are delivered cost-effectively and 
on time, Metrolinx should ensure that contracts 
have firm ceiling prices, whenever possible. 
Contracts should then be monitored for adher- 
ence to the original ceiling price. For work in the 
Union Station Rail Corridor, Metrolinx should 
also consider seeking other qualified suppliers or 
obtaining in-house expertise. 



METROLINX RESPONSE 



Union Station is Canada's busiest passenger 
transportation hub. Balancing operations and 
the safety of the 250,000 passengers who rely 
on it daily while renovating a National Historic 
Site's structure provides unique challenges. 
For example, when renovating an 82-year-old, 
8.6-acre train shed roof, it is difficult to antici- 
pate all structural issues. Co-ordination with 
the federal government and with the City of 
Toronto, who are developing new concourses 
below the train shed, was also challenging. 

When determining procurement options, 
Metrolinx assesses the potential for risk transfer, 
whether the contractor is in a better position to 
manage risks, and the potential cost premium 
for that risk transfer. Metrolinx balances these 
factors to determine the most appropriate 
procurement option under the circumstances, 
recognizing that no contract type is right for all 
projects. At the time of procuring the contract 
for the restoration of the train shed, GO Transit 
determined that the best way to address the 
significant risks associated with this project 
was to use a modified stipulated-price contract. 
GO Transit subsequently engaged an independ- 
ent fairness officer to review the process and 
consider the change orders involved. The review 
confirmed that the process was fair. 

With regard to the Union Station Rail Corri- 
dor, Metrolinx continues to take additional steps 
to reduce its future reliance on existing suppli- 
ers, including obtaining in-house expertise to 
carry out similar work in the future. Metrolinx 
will apply a different model upon the comple- 
tion of the Union Station revitalization, which is 
expected in 2016. 

Double slip switches are a complex section of 
rigid and movable railway tracks more than 50 
metres long and weighing more than 55 tonnes. 
They are very rare in the railway industry, 
and the switches in use at Union Station today 
date back to the 1920s. The renewal program 
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includes upgrading tracks and switches to 
increase their reliability and allow for faster 
train speeds, but replacing this type of switch 
while continuing to operate GO trains was a 
difficult process. With changed construction 
assumptions and the added complexity brought 
on by extensive signal-testing requirements, 
Metrolinx found that switch replacement was 
going to be more time-consuming and expensive 
than we had first estimated. 



REGIONAL TRANSPORTATION PLAN 

Although those GTHA municipalities and transit 
agencies we talked to questioned the priority given 
to some of the RTP's projects, they generally sup- 
ported the plan as currently conceived. One transit 
agency indicated that the plan focuses only on 
new projects and that perhaps some consideration 
should have been given in the plan to maintaining 
existing transit assets. 

Role of Metrolinx 

As noted earlier, one of Metrolinx's key objectives 
is to provide leadership in the co-ordination, plan- 
ning, financing and development of an integrated, 
multi-modal transportation network within the 
GTHA. In order to effectively carry out this man- 
date, Metrolinx's decisions regarding transportation 
and transit planning must be made on the basis 
of a credible business case supported by objective 
and sound data. As well, some of the municipal 
stakeholders we spoke to said that it is important 
for Metrolinx to remain objective and independent 
of any decisions of a political nature made by the 
governments of the municipalities within the GTHA 
and by the federal and provincial governments. 

In the recent debate over the City of Toronto's 
transit projects within the RTP, Metrolinx could 
have been perceived as not being a strong enough 
advocate of what its own analysis suggested was 



the right course of action for these projects. Spe- 
cifically, when the RTP was adopted, the City of 
Toronto's "Transit City" plan — a plan for developing 
public transportation in the city — included light- 
rail transit (LRT) projects on three major arteries 
within the city: Sheppard Avenue East, Finch 
Avenue West and Eglinton Avenue. These three 
projects were adopted into the RTP, and Metrolinx 
prepared benefits case analyses (BCAs) to evaluate 
the costs and benefits of all reasonable alterna- 
tives so that the best version of each project could 
be built, taking cost and service to riders into 
consideration. Metrolinx's analyses concluded that 
the most cost-effective strategy was a mix of light 
rail with traffic on two of the lines (Sheppard and 
Finch) and a fully grade-separated rail system on 
Eglinton Avenue. However, the Eglinton project 
was approved only as a partially grade-separated 
project, because there wasn't enough provincial 
funding for a fully grade-separated system. 

In March 2011, soon after the election of a new 
mayor, the province and Metrolinx entered into 
a Memorandum of Understanding (MOU) with 
the new mayor to revise these projects within the 
RTP. The MOU called for a subway on Sheppard 
Avenue — a decision that would have resulted in 
sunk costs of $65 million — as well as bus rapid 
transit (BRT) on Finch Avenue and a fully grade- 
separated LRT system on Eglinton Avenue. However, 
Metrolinx's analysis concluded that, for Sheppard 
Avenue, the ridership projections did not warrant 
using the higher-cost subway technology along 
the entire corridor. Similarly, forecast ridership for 
Finch Avenue was found to be too large to be effect- 
ively served by BRT. But for Eglinton Avenue, the 
decision reflected in the MOU supported the BCAs 
conclusion that transit/auto conflicts along any 
at-grade sections would affect service reliability on 
the entire line, so that a fully grade-separated LRT 
system would serve the area best. 

Before the MOU was signed, the Sheppard 
Avenue LRT was considered a top priority, and 
construction had already begun. But after the MOU 
was signed, the majority of the provincial funding 
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would now be taken up by the fully grade-separated 
transit line on Eglinton Avenue, so the City of 
Toronto became responsible for funding the Shep- 
pard Avenue and Finch Avenue lines. Because the 
City of Toronto did not have sufficient funds at the 
time to construct a subway on Sheppard Avenue, 
the City initially decided to cease work on the Shep- 
pard Avenue line. The Sheppard Avenue project, 
previously considered a top priority, would now 
be delayed. 

In February 2012, however, Toronto's City 
Council rejected the revised plans under the MOU 
for Sheppard Avenue, Finch Avenue and Eglinton 
Avenue and directed the City Manager to work with 
Metrolinx on developing these projects as previ- 
ously planned — that is, before the MOU was signed. 
On April 25, 2012, Metrolinx formally accepted this 
decision, which will result in approximately $4 mil- 
lion in sunk costs. 

Some GTHA municipalities and transit agencies 
that we talked to used the debate over the City of 
Toronto transit projects as an example to question 
Metrolinx's ability to objectively act as the GTHAs 
central transit planning authority to ensure that the 
most cost-effective and value-added transit infra- 
structure decisions are being undertaken. 



RECOMMENDATION 9 



Metrolinx should ensure that all projects con- 
templated under the Regional Transportation 
Plan are subjected to a rigorous cost/benefit 
analysis that considers financial, economic, 
environmental and social needs and impacts 
and that transit infrastructure investment deci- 
sions are made on the basis of that analysis. 



METROLINX RESPONSI 



Metrolinx has completed a cost/benefit analysis 
on all of the projects included in the first stage 
of the Regional Transportation Plan ("The Big 
Move") . This approach will also be applied 
when considering future projects. 



Plan Funding 

Without long-term sustainable funding, the RTP as 
currently contemplated cannot be implemented. 
The RTP's $50 billion cost estimate may well prove 
low, because it is a high-level estimate derived for 
the most part using average costs per kilometre 
to construct various transit technologies based on 
Canadian and international historical data. The rec- 
ord of cost overruns to date on the priority projects 
we examined also suggests that fully implementing 
the RTP will cost more than estimated. 

When the RTP was adopted, the detailed plan- 
ning and design work that would yield a more 
precise cost estimate had understandably not yet 
begun for the majority of the projects contemplated 
under the plan. Detailed planning and design for 
proposed transit projects can take at least two 
to four years before any construction can begin. 
Although planning and design work is necessary 
before decision makers can be advised on project 
costs and schedule, the costs of doing this necessary 
upfront work may not be fully realized if the project 
is subsequently shelved for a long time due to lack 
of funding. 

Funding has been committed for more than half 
of the priority transit projects within the RTP's first 
15 years. By June 1, 2013, Metrolinx must report 
back to the province on an investment strategy to 
fund the remaining projects within the RTP's first 
15 years, as well as the projects contemplated in 
years 16 through 25 of the RTP. Some examples 
of revenue tools that Metrolinx is contemplating 
using to raise funds for implementing the balance 
of the RTP include an increase in sales and payroll 
taxes specifically for GTHA residents, land value 
capture for GTHA residents whose property value 
has increased or will increase when new transit is 
introduced, and a transit fare surcharge. To this 
end, Metrolinx informed us that one of the key 
issues it faces in formulating a funding strategy is 
identifying the beneficiaries within the region of 
the investment in public transit. 
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Some GTHA municipalities indicated to us 
that Metrolinx has not consulted with them on 
the options being considered. They felt that being 
consulted while the strategy is being formulated — 
specifically on options that will affect local residents 
and businesses (for example, payroll and sales taxes 
specific to GTHA residents and businesses that could 
reduce GTHA competitiveness and job creation) — 
would be beneficial in encouraging timely adoption 
of the investment strategy. Better co-ordination by 
the province and municipalities in formulating strat- 
egies for raising funds would also help avoid dupli- 
cation. For instance, the City of Toronto's January 
2012 report "Sheppard Subway Development and 
Financing Study" identified a number of revenue 
tools that may be available to the city to finance its 
proposed transit expansion. The options anticipated 
were similar to those contemplated by Metrolinx to 
fund the RTP. Metrolinx advised us that it expected 
to start the public consultation phase of developing 
the funding strategy in fall 2012. 

Metrolinx has developed a project prioritization 
framework for ranking unfunded priority projects 
using such criteria as the project's contribution to 
quality of life, environmental health and economic 
prosperity. This framework was not used for 
already funded projects, because those projects 
were approved under previous funding agreements 
and subsequently adopted into the RTP. 

Plan Progress Reporting 

Apart from the timelines covering the first 15 years 
and years 16 through 25, Metrolinx has no other 
defined targets for the overall achievement of the 
RTP. Although there is an urgency to complete 
certain funded projects — such as the Air Rail Link 
and projects associated with the revitalization of 
Union Station — in time for the summer 2015 start 
of the Pan/Parapan American Games, the remain- 
ing projects have no clearly defined timelines for 
completion. As noted earlier, funding dictates the 
completion of these projects for the most part. 



In our discussions with GTHA municipalities, 
some indicated that Metrolinx should more 
regularly update their respective councils on the 
RTP's overall status, including the status of initia- 
tives contemplated under the RTP that are not yet 
funded. These updates would help municipalities 
to better prioritize local projects. For instance, 
one municipality indicated that the impact of 
the uncertainty of funding on the timing of local 
projects under the RTP was making it difficult for 
the municipality to co-ordinate certain of its public 
works infrastructure projects. 

Metrolinx plans to release an update to the RTP 
in June 2013. This update will be an addendum 
to the current plan, and will reflect any new infor- 
mation that has come to light since the plan was 
adopted. However, no changes are expected to the 
RTP's current vision, goals and objectives, policies 
and actions. 



RECOMMENDATION 10 



To ensure that provincial, regional and munici- 
pal stakeholders are kept up to date on the fund- 
ing requirements and progress of the Regional 
Transportation Plan (RTP), Metrolinx should: 

• regularly consult with GTHA municipalities 
and other key stakeholders as the funding 
strategies are being formulated, especially 
on options that affect local residents; and 

• have clearly defined targets for the 
RTP's more significant projects and 
regularly report on costs and progress 
toward completion. 



| METROLINX RESPONSE 




Metrolinx agrees on the importance of regu- 
lar input and consultation on the Regional 
Transportation Plan ("The Big Move") . Since 
the plan was launched, Metrolinx has been 
regularly engaging municipal officials and key 
stakeholders on the RTP and related initiatives. 
Recently, Metrolinx has increased its engage- 
ment with municipal officials (such as chief 
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accounting officers and treasurers) as well as 
transit managers. 

We have also sought input from municipal- 
ities' chief planning officials through regular 
meetings as we move forward with our work on 
the investment strategy. 

Metrolinx is committed to working with our 
partners and stakeholders and will continue to 
engage in regular dialogue with municipal and 
transit-agency representatives. 



OTHER MATTER 

Project Management Information System 

In 2008, Metrolinx purchased a program manage- 
ment system to plan, record and monitor capital 
projects and report project information. The 
system downloads information from Metrolinx's 
procurement and financial systems and also relies 
on input from project managers for budget and 
actual information. 

We noted that in order to effectively monitor 
projects, project managers often supplemented the 
information provided by the system with manual 
spreadsheets maintained outside the system. This 
approach was necessary because the system did 
not have adequate functionality in areas such as 
scheduling and forecasting. Specifically, the system 
has the following limitations: 

• Limited scheduling capability. The system can- 
not support the scheduling of tasks on mul- 
tiple projects, especially if the projects are on 
the same corridor. For example, we noted that 
the system was unable to provide sufficient 
support for the scheduling of interrelated 
tasks on multiple projects on the Georgetown 
South rail corridor. To compensate, project 
managers had to maintain spreadsheets and 
other scheduling tools outside the system to 
effectively manage their projects; and 

• Limited forecasting capability. The system 
allows users to compare only the year-to-date 



budget to actual results. But to facilitate time- 
lier project monitoring, users must be able 
to make such comparisons for shorter terms 
(that is, monthly or quarterly) . 
In addition, information on disbursements 
related to various projects is downloaded to the sys- 
tem from Metrolinx's financial system each night. 
These disbursements are supposed to match the 
commitments set up in the system for the respective 
projects. However, we noted that the system was 
not properly distributing all costs incurred to the 
appropriate commitment. This impacts the project 
managers' ability to reliably compare the projects' 
actual costs to their respective budgets. 



RECOMMENDATION 11 



Metrolinx should ensure that its project man- 
agement information system provides the 
functionality needed to facilitate the effective 
monitoring of individual projects. 

ETROLINX RESPONSE 

Metrolinx will continue to introduce project 
management tools and training to support its 
project managers in ensuring that projects are 
completed on time and on budget. For instance, 
some of the anticipated and implemented 
tools include: 

• project and program dashboards to allow 
project managers and senior management to 
track project performance on key indicators; 

• new functionality to be added to the existing 
project management information system to 
allow for monthly forecasting with associ- 
ated dashboards and reports; and 

• more scheduled training for user access and 
integration with additional project manage- 
ment systems, where there is value added, to 
have greater overall functionality in schedul- 
ing, cost control and risk management. 



Chapter 3 
Section 




Ministry of Community Safety and Correctional Services 



Ontario Provincial Police 



Background 




The Ontario Provincial Police (OPP) operates under 
the Police Services Act (Act), and primarily provides: 

• front-line police services in areas of Ontario 
that do not have their own police force; 

• patrols on provincial highways, waterways 
and trail systems; 

• municipal policing services under contract 
where requested by municipalities; 

• emergency and other support services to all 
communities in the province; and 

• investigations into complex criminal cases and 
organized crime. 

The OPP is one of the largest police forces in 
North America, with about 6,300 police officers 
and 2,300 civilian employees, as illustrated in 
Figure 1. It has 78 police stations, called detach- 
ments, located throughout the province, and the 
detachments operate an additional 87 satellite 
stations. Individual detachments report to one of 
five regional headquarters or to the Highway Safety 
Division, which in turn report to General Headquar- 
ters in Orillia. The OPP also operates five regional 
communications centres that take 911 emergency 
and public calls, and dispatch police. 

The OPP provides municipal policing services to 
322 municipalities on a cost-recovery basis as well 



Figure 1: OPP Full-time Equivalent Staff Strength as of 
May 31, 2012 

Source of data: Ontario Provincial Police 
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Office of the Commissioner 


10 


35 


Corporate Services 


138 


242 


Field Operations 


4,435 


695 


Investigations/Organized Crime 


819 


284 


Traffic Safety and Operational 
Support 


869 


1,058 


Total 


6,271 


2,314 



as to 19 First Nation communities. It also provides 
support as requested to 53 municipal and nine 
Aboriginal police forces of varying sizes in Ontario. 
The Commissioner of the OPP reports to and is 
accountable to the Minister of Community Safety 
and Correctional Services. 

OPP operating expenditures totalled $979 
million in the 2011/12 fiscal year, as detailed in 
Figure 2, with staffing costs representing 87% of 
expenditures. Municipalities receiving OPP policing 
services reimbursed $362 million of that total. The 
OPP also spent $174 million for capital projects, 
such as new facilities for detachments, forensic 
identification units and regional headquarters. 



230 



Ontario Provincial Police 



Figure 2: OPP Expenditures, 2011/12 ($ million) 

Source of data: Public Accounts 
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Field and traffic 
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Audit Objective and Scope 



Our audit objective was to assess whether the 
Ontario Provincial Police had systems and proced- 
ures in place to: 

• ensure its provincial and municipal police 
services are delivered cost-effectively, and 
in compliance with key requirements of the 
Police Services Act and the OPP's own oper- 
ational policies and procedures, known as 
Police Orders; and 

• measure and report on its effectiveness in 
achieving its mandate and priorities. 

Before beginning our fieldwork, we identified 
criteria we would use to address our audit object- 
ive. Senior management at the OPP reviewed these 
criteria and agreed to them. 

The scope of our audit included interviews with 
OPP management, civilian staff and officers; and 
tests, reviews and analysis of relevant files, policies 
and procedures at the OPP's General Headquar- 
ters, three regional headquarters, one regional 
communications centre and 10 detachments. We 
also considered the recommendations we made 
in our 2005 Annual Report and in earlier audits of 



the OPP. As most police work is carried out at local 
detachments, we focused our work on the OPP's 
regional operations. We did not review areas such 
as capital expenditures, investigations and organ- 
ized crime bureaus, and the force's emergency 
preparedness activities. 

We interviewed senior management of the 
Ministry of Community Safety and Correctional 
Services, staff and elected officials from nine 
municipalities for which the OPP provides policing 
services, and representatives from the Association 
of Municipalities of Ontario and the Association 
of Municipal Managers, Clerks and Treasurers of 
Ontario. We researched police operations in sev- 
eral other jurisdictions and visited the RCMP and 
the Surete du Quebec (Quebec's provincial police 
force) to discuss their perspectives on the admin- 
istration of policing services. We also engaged on 
an advisory basis the services of an independent 
expert with senior management experience in the 
policing sector. 

Prior to our last audit in 2005, the OPP did not 
use the government's Community Safety and Cor- 
rectional Services Audit Service Team; however, the 
OPP now utilizes the Ministry's internal auditors 
on a regular basis. We reviewed a number of recent 
internal audit reports and found them helpful in 
planning and conducting our audit, particularly 
their reviews of front-line policing supervision, over- 
time costs and municipal policing arrangements. 

We also found helpful the reports prepared 
by the OPP's Quality Assurance Unit, which 
regularly inspects and reviews police practices at 
detachments. 



Summary 



Over the last two decades, crime rates across 
Canada have declined by more than 40%, and 
Ontario has been part of this trend. For instance, 
since our last audit of the OPP in 2004/05, crime 
rates reported by the OPP have decreased 10%, 
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and serious motor vehicle accidents have also been 
trending down, with fatalities and injuries lower by 
45% and 32% respectively. Over the last five years, 
the average number of calls for service the OPP has 
responded to or initiated has remained relatively 
stable. 

By contrast, OPP expenditures net of recoveries 
from municipalities have increased by 27% over the 
same period. Most of the increase has resulted from 
the hiring of additional officers and staff compen- 
sation increases over the last six years. However, 
our research did indicate that many of the other 
larger police forces across Canada have had similar 
expenditure increases, notwithstanding the long- 
term trend of declining rates of crime and serious 
motor vehicle accidents. 

In our last audit report in 2005, we made a 
number of recommendations on issues relating 
to staff deployment and controlling costs such as 
overtime. We nevertheless found that many of 
these issues continue to exist today. Among our 
major observations: 

• The force has 2,800 officers assigned to 
front-line duties in detachments, but it uses a 
computer-based model developed in the 1980s 
to calculate how to deploy only about 1,250, 
or 45%, of these officers, and only to those 
detachments that provide policing services 
under contract to municipalities. It has been 
more than 10 years since the model was used 
to assess the deployment of some 1,550 other 
officers to detachments that provide other 
front-line policing services, such as patrolling 
provincial highways, or to municipalities that 
use the OPP to provide basic police service 
without contracts. Accordingly, officers face 
significantly different workloads depending 
on where they are assigned. We found that 
officers in some detachments handled an aver- 
age of 54% to 137% more calls than officers in 
other detachments. 

• In 2005, the OPP advised us that it was 
working with the RCMP on a new officer- 
deployment computer model. The OPP has 



since then claimed it uses this new model, but 
it does not. In March 2012, the OPP's existing 
model calculated that the OPP requires 500 
more front-line officers overall, or 18% more, 
whereas the new model calculated that the 
OPP needs 50 fewer officers. We also found 
that the OPP incorrectly applied staffing 
requirements for officer safety in the existing 
model; when the error was corrected, the 
model calculated that only an additional 
156 officers were needed. The OPP also does 
not analyze either its officers' availability to 
immediately respond to a call or the time it 
takes for officers to respond to a call, both 
key factors in determining the deployment of 
officers to detachments across Ontario. 

• The force also lacks a model or method for 
ensuring that 44% of the total detachment 
workforce, comprising the 1,600 officers who 
do not perform front-line duties and 640 
civilian staff, are efficiently and effectively 
deployed. 

• Shift scheduling practices since our last audit 
have shown no improvement whatsoever in 
matching the demand for officers, especially 
during peak workload periods. OPP manage- 
ment had little control over shift scheduling 
at detachments, and almost all officers choose 
to work 12-hour shifts. Current shift arrange- 
ments result in overstaffing during slow 
early-morning hours compared to the busy 
afternoon hours. Similarly, Sundays are over- 
staffed by an estimated 20%. We estimated 
that improved shift scheduling could allow the 
OPP to save millions of dollars per year or, at 
the very least, make its officers more available 
to respond to calls for service. 

• The OPP could assign more corporate-services 
duties and other non-policing work to lower- 
paid civilian staff to free up officers' time for 
policing duties and to save money. The OPP 
has recognized this and initiated a review of 
some staffing positions, which identified more 
than $6 million a year in potential savings. 
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• OPP officers are among the highest compen- 
sated police officers in Canada. Officers and 
civilians receive certain benefits to which 
other members of the Ontario public service, 
such as correctional officers, are not entitled, 
including significantly better pension benefits. 
Since 2003, OPP officers have been receiving 
an annual incentive payment worth up to 
$7,500 to make their salaries comparable to 
another large police force that had negotiated 
a retention incentive at that time. 

• In our 2005 report, we noted that the OPP 
had lowered its overtime costs for 2004/05 
by 10% to $33 million. However, no further 
progress has been made to decrease overtime 
hours, and overtime costs have increased by 
60% to $53 million for 2011/12. 

• The overall cost of OPP services for municipal- 
ities from 2007 to 2011 increased an average 
of 29% for those with contracts and 19% for 
those without, or up to three times the annual 
inflation rate. Municipal officials told us that 
the force offered no adequate explanations 

or accountability for these increases, which 
are due to changes in OPP billing and costing 
methods, and increases in officer compensa- 
tion. On the other hand, all municipal officials 
with whom we spoke said they were very 
satisfied with the OPP services they received. 

• The way the OPP charges municipalities for 
its services was complex and onerous for both 
the force and municipalities. In contrast, the 
RCMP and Surete du Quebec billing processes 
for municipalities appeared much more 
straightforward. 

• OPP processes were insufficient to ensure that 
detachments consistently met the require- 
ments of the Police Services Act and the force's 
own policies. In most of the eight detachments 
we inspected, we noted instances where 
seized cash, weapons and drugs, as well as 
officers' weapons, were either improperly 
stored or accounted for, or were missing. 



• While the OPP reports extensive information 
on its policing activities, crime and motor- 
vehicle accidents in its annual report, it does 
not report information on its own efficiency, 
such as per capita operating costs and 
response times. 



Detailed Audit Observations 



FUNDING LEVELS FOR POLICING 

Over the past two decades, the crime rate across 
Canada has shown a steady decline. In total, 
criminal offences are down more than 40%. In 
Ontario, since our last audit in 2005, the number 
of offences is down about 20%. As well, there has 
been a decline in the number and seriousness of 
crimes reported to police across Canada and in 
Ontario. Statistics Canada's report on the Canadian 
crime severity index, which tracks changes in the 
relative seriousness of police-reported crime from 
year to year, indicates that the severity of crime has 
declined every year since 1991; it was 6% lower in 
2011 than in the previous year and 26% lower than 
in 2001. Across Canada, both the rate and severity 
of violent crime fell 4% in 2011 from the previous 
year, and it was the fifth consecutive annual decline 
in severity of violent crime. In Ontario, the violent 
crime severity index fell 5% from 2010 to 2011, and 
the total crime severity index dropped 6%. 

As well, motor vehicle fatalities and injuries have 
declined by 45% and 32% respectively since 2005. 

A common detachment workload measure for 
policing is "calls for service." More than half of 
all the calls for service that police respond to are 
related to either Criminal Code offences or traffic 
incidents, and they come either from the general 
public or officers themselves. A call from the 
public — a "citizen-generated" call — might involve, 
for example, an assault or a missing person. A call 
initiated by an officer, typically while out on patrol, 
could involve the apprehension of someone driving 
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under the influence of alcohol, although the count 
of officer-initiated calls does not include routine 
vehicle stops for traffic violations such as speeding. 

Over the last five years, the average number 
of all calls for service to the OPP was relatively 
stable at approximately 716,000 a year, with about 
463,000 of them initiated by police themselves and 
253,000 being citizen-generated. 

In total, OPP expenditures have increased 34% 
since our 2005 audit (27% after recoveries from 
municipalities), while the number of calls for ser- 
vice remained unchanged. Increased funding has 
been needed primarily to cover the cost of more 
officers and increases in staff compensation. In con- 
trast, for the six years prior to our 2005 audit, total 
OPP expenditures increased 37% compared to total 
calls for service rising 31%. 

Statistics Canada data indicates the overall 
increase in total OPP expenditures in recent years 
is consistent with the increase in total expenditures 
for all policing in Canada. The number of OPP 
officers rose by nearly 13% from 2004 to 2011, 
comparable to the national increase of 14% over 
the same period. The per capita cost of municipal 
and provincial policing in Canada for 2008 (the last 
year for which the information was published) was 
$268, up from $205 in 2003. Ontario had the high- 
est such cost at $294, up from $225 in 2003. Since 
2005, the OPP has added about 700 officers, 350 of 
whom were hired to fulfill new municipal policing 
requirements for which the cost was recovered. 
Most of the other 350 additional officers were 
approved in 2008, when the province provided 
funding for 200 additional positions for highway 
safety and investigative units, and the federal 
government provided temporary funding for 125 
officers until March 31, 2013. Also, 500 more civil- 
ian employees were added, raising their number by 
28% since 2005. 

In summary, OPP expenditures have clearly 
increased at a much faster pace than inflation, 
even though crime rates and serious motor-vehicle 
accidents are in a trend of long-term decline. We 
do recognize in this regard, however, that expendi- 



tures at other large police forces in Canada have 
increased by similar percentages. And while the 
number of criminal offences reported by the OPP 
from 2005 to 2011 declined by 10%, the clearance 
rate (where charges are laid, or the case is closed 
with no charges) improved by 19%. This clearance 
rate compares favourably to that of other Canadian 
and Ontario police forces. 

We are not aware of any recent independent 
studies or assessments that analyze the long-term 
trend of OPP funding and staffing increases in the 
face of stable or declining demands for police servi- 
ces. However, we understand that the OPP has been 
asked to reduce its expenditures in 2012 to help 
address the province's budget deficit. 



RECOMMENDATION 1 



To support future decisions on funding for the 
Ontario Provincial Police, given the long-term 
trend of decreasing crime rates and fewer 
serious motor-vehicle accidents in Ontario and 
across Canada, the OPP, in conjunction with the 
Ministry of Community Safety and Correctional 
Services, should formally assess the correlation 
of its funding and staffing levels with the actual 
demand for policing services, based on such fac- 
tors as calls for service, motor vehicle fatalities 
and injuries, number of reported offences, clear- 
ance rates for crimes and crime severity levels. 




We agree with the Auditor General that it is 
important to formally assess the correlation of 
our funding and staffing levels with the actual 
demand for policing services. Crime rates are 
not always an accurate barometer of the need 
for police resources, given that proactive work 
and the implementation of prevention programs 
often lead to the reduction of crime. In addition, 
trends can quickly change. The total number of 
deaths on OPP-patrolled roads as of the end of 
September 2012 was 215, compared to 201 at 
the same time last year. 



Ontario Provincial Police 



The OPP is currently in the process of 
updating the Deployment Model to complete an 
assessment of staffing requirements for front- 
line services. The model includes calls for service 
for most criminal offences and all motor-vehicle 
collisions, as well as other parameters used to 
determine front-line policing staffing levels. 

As the province's police service, the OPP 
also provides a wide variety of services not 
identified in the Deployment Model. For 
example, the level of provincial highway staff- 
ing cannot be determined solely on the basis 
of motor- vehicle collision rates; factors such 
as kilometres of highways, traffic volumes 
and patrol standards must also be taken into 
account. Investigations into major crimes and 
organized criminal activity are also influenced 
by a number of factors, including the growing 
complexity of crimes; sophisticated and time- 
consuming investigative techniques; and court 
and case law requirements. Further, assistance 
to municipal police agencies is covered under 
Section 9 of the Police Services Act. Operational 
support and many specialized services (such 
as tactical and emergency response, major 
case management, and search and rescue) are 
staffed based on the need to have these resour- 
ces available to the province in case of major 
incidents — their size cannot be determined 
solely on the number of occurrences. 

The OPP will continue to review its oper- 
ations to find efficiencies and will analyze data 
and information to assess whether savings are 
available or funding can be reallocated across 
programs. Through the Future of Policing Advis- 
ory Committee and other partnerships, the OPP 
will review statistics to ensure that the causal 
factors of crime rates and other trends are 
understood and that decisions are made based 
on sound information. 



MANAGING COSTS 

Detachment Staffing Levels 

The OPP has approximately 4,500 of its 6,300 offi- 
cers and 640 of its 2,300 civilian staff working in 78 
community-based detachments. The rest of its staff 
work primarily at the OPP's regional headquarters, 
communications centres, and at General Headquar- 
ters in Orillia. Individual detachments can have up 
to three distinct policing responsibilities: municipal 
policing for municipalities that have a contract with 
the OPP, basic municipal policing for non-contract 
municipalities, and patrol of provincial highways 
and waterways. 

As we noted in our 2005 audit, the OPP still 
does not have a consistent method for deploying 
officers to detachments. The computer-based model 
that the OPP developed in the 1980s — called the 
Deployment Model — is programmed to calculate 
the number of officers required for front-line duties 
in detachments. The Deployment Model takes into 
account six key parameters: 

• detachment-area characteristics; 

• number and type of citizen-generated calls for 
service; 

• percentage of time an officer is available to 
respond to a call for service; 

• patrol standards; 

• total hours that officers are available for front- 
line policing; and 

• minimum staffing levels for officer safety. 
While the OPP uses the Deployment Model to 

calculate the number of front-line officers required 
in detachments that serve municipalities with con- 
tracts (currently about 1,250 officers), it has been 
more than 10 years since the model was used to 
assess the number of front-line officers assigned to 
other provincial duties or to municipalities without 
contracts (currently about 1,550 officers). The 
lack of a consistent method for deploying officers 
to detachments has contributed to significant vari- 
ances in the workloads of officers depending on the 
detachment to which they are posted. Front-line 
officers in some detachments handle an average of 
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more than twice as many calls for service annually 
as officers in other detachments in the same region, 
as indicated in Figure 3. 

We reviewed the information used by the OPP 
for the parameters of the Deployment Model and 
identified the following regarding its accuracy: 

• In early 2012, the OPP adjusted the model to 
use more current actual workload information 
for officer availability and patrol standards. 
For instance, information for total hours that 
officers are available for front-line policing 
had not been updated since 1998. The annual 
average available hours per officer has 
declined in recent years by 120 hours, from 
1,467 to 1,347, due primarily to additional 
sick time off (22 more hours), administrative 
requirements (53 more hours) and training 
(34 more hours) . As a result, the model will 
in future calculate a need for more front-line 
officers at detachments. Similarly, the new 
traffic volumes, which were last updated in 
the 1990s, will require more officers to meet 
the patrol standards. 

• We also determined that the parameter 
designed to ensure officer safety was 
incorrectly applied. The OPP's policy requires 
that officers work in pairs for the eight hours 
between 10 p.m. and 6 a.m. when practical. 
However, the model used a 12-hour period 
for pairing. Our request for the OPP to re-run 
the model based on the policy requirement 
resulted in a calculation that the OPP required 
only 156 more officers province-wide, or 344 
fewer than the 500 originally calculated, as 
discussed below. However, our interviews 
indicated that, in practice, very few officers 

in detachments actually work in pairs as per 
the policy. As a result, the model has been 
calculating for some time that more officers 
are necessary than actually are required for 
serving municipalities with contracts. 

• Under the model, there is supposed to be an 
officer available to immediately respond to 
any citizen-generated call for service 92% of 



Figure 3: Range in Annual Average Calls for Service 
Per Officer Within Regions (by Detachments), 2011 

Source of data: Ontario Provincial Police 
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the time. However, the OPP does not track its 
success in meeting this target, or whether the 
standard is even reasonable and appropriate, 
because it monitors neither officers' availabil- 
ity to respond to calls nor the average time it 
takes to respond to a call. 
In our 2005 audit, we noted that the OPP 
was working on a new joint OPP-RCMP staff- 
ing model — called the Policing Resource Model 
(PRM) — to replace the Deployment Model. The 
PRM uses information that is similar to but more 
detailed than the information used by the Deploy- 
ment Model to determine how many officers a 
detachment should have for front-line duties. When 
we did our 2007 follow-up report, OPP officials told 
us they had completed testing of the PRM for six 
detachments and that further testing for another 25 
detachments was scheduled for the fall of 2007. The 
OPP told us that if testing was successful, the PRM 
would be implemented across the province. 

However, we found during our current audit 
that the OPP does not use the PRM for determining 
the number of front-line officers required; instead 
it continues to use the older Deployment Model. 
Despite this, the OPP said in its 2008 business plan 
to the Ministry of Community Safety and Cor- 
rectional Services (Ministry) that it was using the 
PRM, and in 2012 the Ministry informed Treasury 
Board that, based on the PRM, the OPP had a short- 
fall of officers to support front-line service delivery. 
The OPP even received an international policing 
award in 2009 for its implementation of the PRM. 
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We noted that the RCMP is currently using the PRM 
and has been doing so for more than five years. 

In March 2012, the OPP conducted an analysis 
of its front-line officer requirements at all detach- 
ments using both models. The Deployment Model 
calculated that the OPP required 500 more officers, 
whereas the PRM calculated that it actually needed 
50 fewer officers. In addition, both models identi- 
fied that detachments required anywhere from 
30% fewer to over 100% more officers, although 
the results varied widely between the models. For 
example, at a detachment with 59 officers, the PRM 
calculated that 28% fewer officers were needed 
while the Deployment Model calculated 105% more 
officers were required. We talked to a number of 
regional and detachment senior staff, including 
staff in the 59-member detachment above, and they 
indicated to us that, from an operational perspec- 
tive, they had no serious concerns with their cur- 
rent officer staffing levels. 

Both staffing models calculate the number of 
front-line officers needed at detachments, but the 
OPP has not established a staffing template for 
the other 44% of staff who also work in detach- 
ments, including 1,600 officers who do not perform 
front-line duties and 640 civilian staff. These other 
officers work in administrative and other policing 
duties, including courts, community relations and 
crime investigations. We could find no basis for 
determining the number of these other officers — 
for example, no detachment-workload indicators 
were being used. As well, we were told that the 
number of officers and civilians working at detach- 
ments was generally based on historical levels, and 
the numbers have been relatively stable over the 
last five years. 

At the time of our audit, the OPP was engaged in 
a project to develop templates for detachment staff- 
ing and to identify inconsistencies and inequities in 
current detachment staffing levels. 



RECOMMENDATION 2 



To help ensure that the number of front-line 
officers at each detachment is based primarily 
on need and that officers are cost-effectively 
deployed, the Ontario Provincial Police should 
reassess its two computer-based models to 
determine which one provides the best estimate 
of requirements based on up-to-date and accur- 
ate operational and workload standards, and, 
once validated, use its staffing models to deploy 
and reassign officers. The OPP should also 
establish formal staffing methodologies for the 
other 44% of detachment staff not covered by its 
deployment model. 

| 0PPRESP( 

The OPP has conducted extensive analysis 
comparing the Policing Resource Model (PRM) 
and the Deployment Model, and determined 
that the Deployment Model is the most suitable 
model for identifying municipal and provincial 
staffing requirements at detachments. The OPP 
is updating the Deployment Model parameters 
and expects to complete this project in late 
2013. This will make it possible to compare 
staffing results from the model with the actual 
staffing levels at each detachment. Given the 
diverse nature of detachments, local input and 
analysis of operations must also be conducted 
and taken into account to determine the 
approved staffing requirements. 

The OPP agrees with the Auditor General 
that formal staffing methodologies are neces- 
sary for the detachment staff not covered by the 
Deployment Model, such as specialists, admin- 
istration and supervisory positions. The OPP 
has been working on a set of consistent methods 
for staffing the various types of non-front-line 
staff at detachments. The detachment structure 
review has led to the creation of a model for 
ideal staffing of non-front-line staff, which the 
OPP will implement as part of its current review 
of staffing levels in detachments. 
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Officer Shift Scheduling 

As with staff deployment, a key to operating cost- 
effectively is having the right number of officers 
working at the right times. In 1998 and in 2005, 
we reported that officers were not scheduled in a 
way that corresponded to peak workload periods, 
and this issue remains. We saw no evidence of any 
changes to make more efficient use of staff time 
in order to reduce staffing costs or improve officer 
availability during peak times. 

We learned in interviews that OPP management 
had little control over shift scheduling at detach- 
ments. Instead, local officers generally determined 
their own scheduling, and almost all chose to work 
12-hour shifts, which offer more days off. Manage- 
ment in detachments whom we spoke to said they 
believed they did not have the authority to change 
officers' 12-hour shift schedules, and it was appar- 
ent that there were different interpretations and 
knowledge around OPP shift-scheduling policies 
and the collective agreement. 

For the most part, the 12-hour shifts are sched- 
uled without overlap, so that the same number of 
officers is working at all times during the day and 
all through the week. Alternatively, overlapping 
shifts can offer the benefit of having more officers 
working during the busiest periods of the week and 
day. Although overlapping shifts could pose a prob- 
lem if, for example, there were not enough avail- 
able patrol vehicles, the issue has not been formally 
examined to assess its workability. 

At detachments, the total number of calls for 
service and citizen-generated calls for service from 
3 a.m. to 7 a.m. is only about one-third of the num- 
ber from 3 p.m. to 7 p.m., when demand is greatest, 
as shown in Figure 4. Because the scheduling of 
shifts is not based on workloads, detachments are 
overstaffed during slow times in the early morning 
hours compared to the busy afternoon hours. Front- 
line officers told us that after 3 a.m., they spend 
most of their time catching up on writing reports 
and court briefs, and other administrative duties. 
However, we also noted that the OPP has already 
reduced the burden of report-writing at many 



detachments by assigning more of those duties to 
civilian staff. 

OPP senior management acknowledged that 
moving from 12-hour shifts to a mixture of shift 
lengths, using part-time officers to supplement 
regular officers and increasing shift overlaps dur- 
ing peak times would save money and improve the 
productivity of detachments. However, manage- 
ment also advised us that the current collective 
agreement might not contain the flexibility needed 
to make such changes. We estimate that adjusting 
the current 24-hour fixed scheduling from equally 
manned 12-hour shifts to a mixture of shifts and less 
staff from 3 a.m. to 7 a.m. would either result in sav- 
ings in the range of $5-10 million a year or lead to 
improved service to the public and officer safety by 
more effectively matching staffing to workload. 

In addition, the OPP operated 77 of its 78 
detachments on a 24-hour basis in 2011. In 2008, 
only 68 of the detachments provided policing 
24 hours a day; the other 10 were open 20 hours 
a day, with officers working 10-hour shifts. Even 
though the number of calls for service remained 
essentially unchanged from 2008, the OPP now 
operates virtually all its detachments on a 24-hour 
basis, with almost all of its officers working 12-hour 
shifts. For instance, citizen-generated calls for ser- 
vice averaged 1,237 per year over the past five years 
for its only detachment open for only 20 hours a 
day. Yet, we identified eight detachments that were 
open 24 hours a day and had either the same or 
even significantly fewer citizen-generated calls for 
service. In 2012, the OPP did change one detach- 
ment's hours so that it also operates only 20 hours 
per day. 

The same issue exists with respect to weekday 
versus weekend shift scheduling. The schedules 
are made up so that the same number of officers 
is assigned to each day of the week. We calculated 
that on Sundays, there was about 20% overstaffing 
of officers, based on the average number of 
citizen-generated calls for service. By redeploying 
officers throughout the week based on the demand 
for service, the OPP could either reduce costs by 
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Figure 4: Total Number of Calls for Service by Hour of Day, 2011 

Source of data: Ontario Provincial Police 
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several million dollars or make more effective use 
of those officers. 

Shift scheduling also becomes a significant issue 
in the summer months, when the demand for OPP 
service grows in many areas of Ontario with the 
influx of tourists and cottagers. Figure 5 shows that 
citizen-generated calls for service were 11% higher 
than the 12-month average during the summer 
months of July and August in 2011; however, it also 
shows that officers take significantly more vacation 
in these months than in any other month. In fact, 
total officer hours worked during the months of 
July and August were less than most other months 
that had fewer calls for service. For example, 8% 
more staff hours were worked in March than in 
July, yet there were 23% more citizen-generated 
calls for service in July. In addition, we estimated 
that the OPP paid $2 million in overtime to cover 
officers' summer vacations. 



We raised this same concern in our 1990 and 
1998 reports. In 1990, front-line officers were 
taking 40% of their collective vacation during 
the summer months, and in 2011, that figure was 
about 35%. Some police forces have separate 
summer and winter work schedules to make sure 
they have enough staff in the summer, but the OPP 
has yet to implement this practice. We noted that, 
while the OPP has a policy that requires detach- 
ment management — when they approve vacation 
time — to ensure a minimum number of officers 
be maintained during the summer months, the 
expected calls for service are not formally taken 
into consideration. As well, since the population 
in many areas serviced by the OPP swells in the 
summer months and the number of calls for service 
increases, consideration should be given to increas- 
ing staff availability rather than maintaining min- 
imum staffing levels at detachments. 
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Figure 5: Monthly Percentage Change in Averages for 
Four Variables, 2011 

Source of data: Ontario Provincial Police 
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In 2000, in response to our 1998 recommen- 
dations, the OPP introduced a shift-scheduling 
manual. However, by 2008, only a few detachment 
commanders had used the manual to change their 
shift schedules. As a result, in 2009, the OPP issued 
a new policy requiring that detachment and regional 
management annually review shift schedules. The 
policy also required that a committee be established 
to conduct an annual review of shift scheduling and 
issue a report to senior OPP management by August 
of every year. However, there have been no formal 
reviews of scheduling practices at any level, and the 
committee has not yet met. 



RECOMMENDATION 3 



In order to reduce operating costs and/or make 
the best use of available officers' time to more 
effectively respond to calls for service, the 
Ontario Provincial Police should: 

• implement measures to give management 
greater control over officers' shift schedul- 
ing and vacation entitlements to better 
co-ordinate staffing with hourly, daily and 
monthly demand for policing services; 

• provide detachment management with 
regular information that compares work- 



load with staffing levels during all times of 
the year; 

reassess its current practice of having almost 
all detachments operate on a 24-hour basis 
to identify the savings potential of reducing 
operating hours at some detachments; and 
monitor adherence to the existing policy 
requirement that shift scheduling practi- 
ces at detachments be formally reviewed 
annually to assess their efficiency and cost 
effectiveness. 



The OPP recognizes that shift scheduling is 
an important operational issue with a formal 
protocol outlined through the collective 
agreement. Significant research on this has 
been undertaken by the OPP in the past year. 
The OPP considers operational issues such 
as vehicle requirements, supervision, officer 
health and morale, training, court appearances, 
administrative work and report-writing. Due to 
operational considerations as well as the rural 
nature of OPP detachments, more formal assess- 
ments of shift scheduling at each detachment 
may result in identifying savings that are less 
than the figures estimated by the Office of the 
Auditor General and lower than those that an 
urban police service might generate. In addi- 
tion, the OPP is reviewing its policies regarding 
24-hour detachments to assess the viability of 
this type of service in some locations. 

The OPP will continue to seek and imple- 
ment measures to better co-ordinate staffing 
with the demand for police services. Several 
tools and types of training have been developed 
and are available for detachment command- 
ers on the OPP intranet. The OPP's goal is to 
develop additional tools, such as a software 
tool in 2013, for detachment commanders to 
better analyze the implications of existing and 
proposed schedules, consistent with the criteria 
that must be considered as part of the process. 
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When the comprehensive analytical tools are 
finalized, it is anticipated that detachment man- 
agement and staff will work together to develop 
their own improved staffing schedules. Each 
region will conduct annual reviews to ensure 
optimum shift scheduling. 

The OPP has made efforts to include shift 
scheduling in collective agreement negotiations. 
This was not successful in the last round of bar- 
gaining; however, the OPP continues to press for 
its inclusion. Shift scheduling changes using the 
existing manual will require a lengthy timeline 
and a detachment-by-detachment review and 
approval process. Shift scheduling is included 
in our proposed transformational plans and sav- 
ings are expected by 2014/15. 



Use of Civilians 

The OPP could make better use of more civilians, 
who already make up 27% of its staff, to free up 
officers for policing duties and reduce operating 
costs. We noted this issue as far back as 1990. 

In early 2012, the OPP said it determined that 
having civilian employees take over more court and 
community-service duties now performed by offi- 
cers would lower staffing costs. As of 2012, there 
were 103 officers working in such community- 
service duties as crime prevention programs in 
schools and media briefings. A first-class officer 
earns a salary of $83,500 a year, while a civilian 
community officer earns $57,250. The overall sav- 
ings if all these duties were transferred to civilians 
would be about $2.6 million a year. Similarly, there 
are about 110 officers and 23 civilians performing 
court duties. Transferring all court duties to civil- 
ians would save another $2.8 million annually. 

In its recent cost-management study, the OPP 
identified positions in Corporate Services now filled 
by officers — in areas such as business, finance, 
human resources, training, records management, 
policy development, and equipment and fleet man- 



agement — that could just as well be filled by civil- 
ians. The study said the OPP could save $760,000 
by shifting some of these jobs to civilians. 

We noted that the OPP has not done a full 
review to identify the jobs where officers could be 
replaced by qualified civilian staff, apart from the 
positions already mentioned. In many cases, offi- 
cers are assigned to administrative positions while 
retaining their officer titles and salaries and bene- 
fits, and they continue to receive the same police 
training as front-line officers. The RCMP advised us 
that they introduced a cost-saving initiative in 2012 
to more rigorously ensure officers are used only for 
policing activities, while civilians handle virtually 
all other non-policing and administrative duties. 



RECOMMENDATION 4 



To help ensure that non-policing duties and 
responsibilities are handled as cost-effectively as 
possible, the Ontario Provincial Police should: 

• conduct a review of all staffing positions and 
responsibilities at its detachments and its 
regional headquarters and General Head- 
quarters to determine where opportunities 
exist to fill positions currently held by offi- 
cers with civilians at a lower cost; 

• establish cost-saving targets and timelines 
for designating positions to either civilians 
or officers, depending on the duties of the 
position; and 

• reassign officers who are currently in civilian 
positions back to front-line policing duties 
where possible. 

OPP RESPONSE 

The OPP is a participant in the Future of Poli- 
cing Committee (FPAC) review created by the 
Ministry of Community Safety and Correctional 
Services and involving municipal and police 
stakeholders to redefine core duties. This 
extensive review will provide service options 
for consideration by all Ontario police agencies. 
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The OPP has conducted a preliminary analysis 
of potential savings opportunities with regard 
to civilianization and will complete the analysis 
with a more comprehensive review of the cost 
savings and the operational impacts. Due to 
provincial responsibilities and the number of 
small, geographically dispersed detachments, 
the OPP must ensure it has adequate police offi- 
cers available for local emergencies and major 
provincial events such as Caledonia, G8/G20, 
major investigations and others. 

The OPP acknowledges that with the finan- 
cial constraints in today's economy, the cost 
savings of increased civilianization may be a 
component in the future of policing. The OPP is 
committed to continuing its review of staffing 
positions and will develop cost-savings targets 
and timelines for changes. 

The OPP is limited by the parameters of 
the collective agreement in relation to the 
establishment of and changes to job descrip- 
tions and classifications. Some roles currently 
being performed by uniformed officers that 
could potentially be civilianized (e.g., Com- 
munity Service Officers) do not have specific job 
descriptions associated with them because they 
are not distinct positions. In addition, there are 
many operational benefits of police performing 
Community Service Officer and Court Services 
Officer roles: they are able to respond to other 
calls for service, they have powers of arrest, 
and they generally have more influence over 
students and the public. 



Differential Response Unit 

The OPP uses what it calls a Differential Response 
Unit (DRU) in each of its five regions and its High- 
way Safety Division to handle low-priority calls for 
service over the phone. The goal of the program is 
to make the most efficient and cost-effective use of 
front-line officers' time. However, as we noted in 



our 1998 and 2005 audits, the DRU program is not 
yet consistently managed and executed. 

The DRU function is supposed to work as fol- 
lows: When any of the five regional communica- 
tions centres receives a citizen-generated call for 
service, the detachment platoon sergeant decides 
whether an officer will be sent out or if the call can 
be handled over the telephone at a later time. If 
the sergeant decides that a front-line officer does 
not need to be dispatched, the call is forwarded to 
the region's DRU, which normally deals with the 
call over the phone. (Under OPP policy, however, 
if a caller or a municipality asks for an officer to 
respond in person, one is dispatched regardless of 
the nature of the call.) 

The OPP reviewed DRU operations in 2007 and 
found that all regions did not provide the same 
kind of service and the units often lacked proper 
supervision, which could cause service delivery to 
suffer. The OPP noted these inconsistencies and 
deficiencies again in 2010, and said it would launch 
a project to ensure that the DRU program had a 
standardized structure to ensure consistent manage- 
ment and accountability. However, we found no evi- 
dence that this project was ever set up. OPP senior 
command requested another review of the DRU pro- 
gram, to start in fall 2011. We were informed that as 
of July 2012, a draft report was being finalized. 

Our review of the number of calls handled by 
DRUs in 2011 noted significant differences in use of 
the program, as shown in Figure 6. One region, for 
example, used DRU to handle just 1% of its calls. 

We believe that the DRU program, with proper 
management, could help the OPP reduce operating 
costs and make more efficient use of the time of its 
front-line officers. However, only one region has a 
centralized DRU; in the others, local detachments 
cover DRU functions by assigning the duties to 
officers being accommodated for pregnancy, injury, 
illness or any other cause making them unable to 
go out to respond to calls. It is likely more cost- 
effective and efficient for a region to centralize its 
DRU or for all communications centres to share one 
DRU for the entire province. 
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Figure 6: Percentage of Citizen-generated Calls for 
Service Handled by Differential Response Units, 2011 

Source of data: Ontario Provincial Police 



Region 


% of Calls 


Central 


10 


East 


8 


North East 


1 


North West 


22 


West 


6 


Highway Safety* 


Unknown 



* The Highway Safety Division operates a DRU, 
but does not maintain any statistics on its use. 



In addition, in our 2005 Annual Report we noted 
that one region's DRU officers were able to each 
handle an average of 950 calls for service annually, 
compared to about 200 calls for a front-line officer. 
In the current audit, the same region reported that 
its DRU officers each handled on average fewer 
than 600 calls per year, and consequently, if more 
calls were forwarded to DRU, they could likely be 
handled with minimal staffing impact. 



RECOMMENDATION 5 



To help achieve the significant cost and oper- 
ational benefits of implementing a Differential 
Response Unit (DRU) program to free up front- 
line officers' time for more serious matters, the 
Ontario Provincial Police should: 

• establish a strategic plan for fully imple- 
menting its DRU program throughout the 
province, with targets for measurable savings 
and benefits, and associated timelines; and 

• given the lack of widespread success in 
implementing the DRU program over the 
last decade, consider centralizing the 
program to improve service levels, enhance 
consistency and help realize economies of 
scale and cost savings. 

| 0PP RESPONSE 

The OPP acknowledges that there are oppor- 
tunities to enhance and improve the Differential 



Response Unit (DRU) program. A comprehen- 
sive report, reaffirming these opportunities, was 
completed and presented to senior management 
in September 2012. 

All regions have established a DRU program 
to some extent, but implementation provincially 
has been inconsistent, as it relates to having an 
effective operating environment and staffing 
requirements, including short-term accom- 
modated officers, and appropriate management 
and training. The current report concludes that, 
rather than centralization, the most decisive suc- 
cess factors are having clear direction from the 
regional commanders, oversight, accountability 
and well-defined reporting relationships among 
those assigned to perform the DRU function. 

The OPP is committed to moving forward 
on this initiative and will invest the resources 
required to implement improvements, includ- 
ing setting performance targets and measuring 
against these targets. 



OFFICER COMPENSATION 

Salary and Benefits 

According to a quarterly RCMP survey of salaries, 
benefits and pensions, the OPP ranked third among 
Canadian police forces at $66.45 an hour as of 
March 31, 2012. Only the Toronto Police Service 
($70.90/hour) and the Vancouver police ($70.52/ 
hour) were higher. The RCMP stood at $65.72 an 
hour and the Surete du Quebec at $57.84. 

OPP members received an average annual 
salary increase of 2.9% for 2005 to 2010, a 5% 
wage increase in 2011, and no increases for 2012 
and 2013. However, following a recent provincial 
government announcement that OPP officers would 
receive the "highest paid first-class constable base 
rate" in the province, an officer's annual salary may 
well increase by at least 8.5% effective January 
1, 2014. In addition, officers and civilian staff are 
entitled to a number of compensation benefits that 
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few other members of the Ontario public service, 
such as correctional officers, receive. For instance: 

• In 2003, after the Toronto Police Service 
began paying its officers a retention bonus to 
encourage them not to leave the force in sig- 
nificant numbers, the provincial government 
negotiated an incentive called the Provincial 
Responsibility Incentive with the officers' 
union — even though staff were not leaving 
the OPP for other employment in significant 
numbers — that resulted in OPP officers being 
paid at compensation levels comparable 

to Toronto officers. Commissioned officers 
(generally senior OPP management staff) 
who are not part of the union also received 
the incentive, which was officially established 
to compensate officers for being expected to 
respond across the province to emergencies, 
for long-term investigations and for large local 
events when extra resources are required. In 
2012, the annual incentive was 3% of salary 
after eight years of service; 6% of salary after 
16 years of service; and 9% of salary for 23 
years or more, up to a maximum of $7,500. 
We noted that in 2002, before the incentive 
was provided, only 43 officers that year left 
the force, which then stood at 5,300. In 2011, 
only 25 officers left the force. 

• OPP staff have better pension benefits, includ- 
ing a lower retirement age, than other Ontario 
government employees who are members of 
the same Public Service Pension Plan or other 
pension plans. This in itself is a retention 
incentive. In addition, as of January, 2012, the 
pensions of officers are calculated based on 

a 36-month average of their highest annual 
salary, rather than a 60-month average used 
for other provincial government employees. 
For civilian OPP staff, the pension is based on 
a 48-month average. Using fewer months to 
determine average salaries results in higher 
pension costs for the government. 

• Each year, instead of taking five vacation days, 
staff may opt to receive an additional week's 



pay. From 2009 to 2011, 30% of staff received 
this payment, and it represented about $4 mil- 
lion annually. 
• The OPP provides an allowance of $1,250 
annually to each officer whose policing duties 
require plain clothes in addition to uniforms. 
In 2006, the OPP extended that allowance 
to all 200 commissioned officers, who work 
primarily in an office environment, where 
the OPP has a business-attire dress code. In 
2010, the allowance was expanded to include 
dry cleaning expenses. The cost of providing 
commissioned officers a clothing allowance is 
$250,000 annually. 

Overtime 

In our 2005 Annual Report, we noted that the OPP 
had achieved some success in lowering overtime 
expenditures, decreasing them by more than 10% 
to approximately $33 million in 2004/05. Since 
then, overtime hours have been kept to a consistent 
rate per officer, but the impact of compensation 
increases and increases in the number of officers has 
been a 60% increase in overtime costs to $53 mil- 
lion for 2011/12. Approximately 63% of overtime 
was incurred by officers in detachments, who on 
average received overtime payments of $7,500 each 
in the 2011/12 fiscal year. The increase in overtime 
occurred even though over the last five years, the 
average annual total calls for services handled by 
the OPP remained relatively stable, the number of 
criminal offences reported by the OPP declined and 
the OPP has added 700 officers since 2007. 

In addition, we noted that in 2011, about $7 mil- 
lion of overtime was logged for administrative dut- 
ies, such as training, report-writing and meetings, 
in apparent contradiction of OPP overtime policy. 



RECOMMENDATION 6 



To help inform future decisions on compen- 
sation levels for officers and as part of the 
preparation for future collective bargaining 
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negotiations, the Ontario Provincial Police 
(OPP) should analyze the working condi- 
tions and compensation levels of its officers in 
comparison to other major police forces across 
Canada and in relation to current Ontario gov- 
ernment compensation policies. 

The OPP should also increase its oversight 
of overtime expenditures as well as identify and 
address the underlying reasons for the signifi- 
cant increase in overtime costs in recent years. 



| OPP RESPONSE 

The OPP has committed in its 2012 trans- 
formational plan proposal to continue to work 
through the Police Bargaining Working Group 
to identify opportunities to bend the cost curve 
in police compensation. The OPP uses Statistics 
Canada and RCMP Police Sector Total Compen- 
sation reports and communicates with other 
policing agencies regarding compensation. The 
Employer (Ministry of Government Services), 
in consultation with OPP, has reviewed many 
aspects of OPP compensation and will continue 
to bring these forward to the negotiation table. 
The OPP will continue to assess compensation 
trends and will annually review its compensa- 
tion compared to other policing agencies and 
the Ontario government. 

The OPP agrees that overtime is an import- 
ant cost to manage. After significant effort, 
overtime costs were brought to very low levels 
for the 2004/05 fiscal year, as noted by the Aud- 
itor General. Since that time, overtime has been 
significantly impacted by major events including 
protracted First Nations protests, several large- 
scale investigations and the G8/G20. 

Although overtime has increased in dol- 
lar terms, a portion of the increase is directly 
attributable to compensation increases and the 
increase in officers for new municipal contracts. 
Administrative overtime is generally not permit- 
ted and in some cases may be a coding issue 
rather than administrative overtime. The OPP 



will review the current administrative overtime 
and will ensure it is reduced where appropri- 
ate or coded to the correct activity. The results 
will be monitored and reviewed for 2012/13 
and onward. 

Over the past few years the OPP imple- 
mented additional overtime oversight and 
processes including detailed monthly overtime 
reports, detailed detachment analysis and 
memos regarding overtime management. Over- 
time management is part of the OPP's current 
cost management strategies and savings targets 
are reflected in the Results-based Planning 
process, underlying the OPP's commitment to 
overtime management. 



MUNICIPAL POLICING 

Under the Police Services Act (Act), municipalities 
may establish police forces either on their own or 
with other municipalities, or they may purchase 
policing services from other municipalities or from 
the OPP under a five-year renewable contract with 
the Ministry. If municipalities make no arrange- 
ments, the OPP is required to provide the basic 
policing services it thinks are necessary and those 
municipalities reimburse the OPP for the cost of 
the policing provided. Municipalities that have 
contracts with the OPP have more control over the 
accountability and level of service that they get 
from the OPP, and in setting community priorities 
for their own policing. 

During the 2011/12 fiscal year, the OPP serviced 
149 municipalities with contracts, and 173 muni- 
cipalities that pay for services without a contract. 
The OPP received $362 million in revenues for 
providing these services, with 75% of that coming 
from municipalities with contracts. Billings and 
collections for both contract and non-contract 
municipalities are handled by the Shared Services 
Bureau of the Ministry of Government Services 
using information provided by the OPP. In addi- 
tion, the provincial government assists small and 
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rural municipalities with their policing costs. In 
2011, municipalities policed by the OPP received 
$71.9 million from the Ministry of Finance under 
the Police Services Grant Program. 

In addition, the OPP provides support to smaller 
and rural municipal police forces that may need 
assistance from time to time, and to help them 
meet policing standards for events such as emer- 
gency response. 

Relations with Municipalities 

A number of municipal officials told us that while 
their annual costs for OPP services have increased 
significantly, the OPP offered no adequate explana- 
tions or accountability for these increases, and 
showed no flexibility to limit them. The OPP and 
Ministry have acknowledged these concerns and, 
during our audit, were working on ways to better 
share information to improve the relationship with 
municipalities. 

Municipalities are paying more for OPP services 
largely because of changes over the last several 
years to the OPP's billing and costing methods, as 
well as increases in officers' salaries. We calculated 
that for municipalities with contracts, the overall 
cost of the OPP's services increased 29% from 2007 
to 2011, an average of about 7% annually — or more 
than three times the rate of inflation. For municipal- 
ities without contracts, the overall increases were 
19% over the same period, or an average of about 
4% annually. Over the same period, individual 
municipalities experienced a range of changes in 
costs for OPP service, from a decrease of 60% to an 
increase of 135%. Factors besides internal OPP cost 
increases that affect the chargeback include changes 
in the number of calls for service to a municipality. 

Notwithstanding the concerns expressed about 
costs, all municipal officials we spoke with said 
they were very satisfied with the OPP services they 
received and with the communication provided 
by local detachment commanders on community 
issues. As well, all the municipalities acknowledged 
that using the OPP's services was generally less 



costly than maintaining their own municipal police 
forces. According to OPP estimates, municipalities 
with and without contracts save on average any- 
where from 35% to 60% by using the OPP instead of 
having their own police forces, as shown in Figure 7. 

OPP senior staff told us that some municipalities 
choose not to sign contracts for OPP services, while 
others with contracts decide to pay for services over 
and above the basic requirements. Consequently, 
municipalities can receive different levels of poli- 
cing. As an example, two municipalities with simi- 
lar numbers of calls for service had very different 
levels of service: One, a non-contract municipality, 
paid $9 million a year for a minimum level of police 
service; the other paid more than $15 million a year 
for a contract that provided for 30 staffing enhance- 
ments, including a dedicated traffic patrol and 
greater officer presence in the community. Several 
larger municipalities have chosen a basic level of 
service by not entering into contracts with the OPP, 
likely due to the lower costs of such arrangements. 
Interestingly, the OPP advised us that three muni- 
cipalities that paid for enhanced OPP services were 
cited in 2010 by a national magazine as being in the 
top five safest communities in Canada. 

The Ministry and OPP have been working 
to improve relationships with municipalities by 
providing more information on the costs of their 
services. In February 2012 , a project team of 
representatives from the OPP, the Ministry, munici- 
palities and their associations, and police services 
boards was formed to improve communication 
and accountability about the OPP's costing policies 
and processes, and to identify opportunities for 
municipalities to contain or reduce policing costs. 
In March 2012, about 40 policing organizations 
and municipal stakeholders took part in a summit 
on the future of policing hosted by the Ministry. As 
well, the Ministry created the Future of Policing 
Advisory Committee with municipal involvement 
to oversee and make recommendations by 2015 for 
changes to policing standards. 
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Figure 7: Cost Comparison of Municipal Police Services, 2011 

Source of data: Ontario Provincial Police 







Estimated per Capita Cost 
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Population 100,000 and over 


Municipal police force 


17 


282 


OPP - with contract 







OPP - no contract 







Population 50,000-99,999 


Municipal police force 


7 


264 


OPP - with contract 


6 


153 


OPP - no contract 


2 


117 


Population 15,000-49,999 


Municipal police force 


14 


284 


OPP - with contract 


21 


150 


OPP - no contract 


4 


131 


Population 5,000-14,999 


Municipal police force 


14 


329 


OPP - with contract 


48 


156 


OPP - no contract 


31 


133 


Population less than 5,000 


Municipal police force 


1 


371 


OPP - with contract 


74 


238 


OPP - no contract 


136 


142 



Costing and Billing Methodology 

The OPP sets the minimum number of officers and 
hours required to service a municipality, regardless 
of whether it has a contract in place. This deter- 
mines the municipality's expected costs, including 
direct costs for officers' salaries plus an allocation 
for other support and operating costs, such as 
vehicles, training, communications, forensic iden- 
tification units and facilities. Municipalities with 
contracts can also opt to enhance the service level 
beyond the minimum recommended by the OPP, 
and to pay extra for more officers. Municipalities 
are given a budget at the beginning of the year, 
billed monthly based on that budget, and receive 
a final invoice that reconciles the budget to actual 
expenses at year-end. 



To determine the minimum number of officers 
a municipality requires, the OPP uses either its 
Deployment Model or a workload analysis tool. 
However, these different methods can produce 
different results. In 2011, internal auditors noted 
that the OPP had not established clear criteria for 
using one or the other method. We were told that 
a manual on the contract renewal and negotiation 
process was to be completed by late 2012, and 
shared with municipalities, to address the internal 
auditor's recommendation that the OPP ensure that 
the process of establishing municipal contracts is 
transparent and managed effectively. 

As of our current audit, the agreements with 
municipalities did not clearly indicate whether 
the number of officers was the original minimum 
determined by the OPP or whether it was a higher 
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number based on a municipal request. For example, 
contracts for one municipality showed that the 
number of officers increased from 65 in 2009 to 80 
in 2013. However, there was no indication whether 
these additional officers were needed as a minimum 
to meet increasing workloads, or whether they had 
been requested as enhancements to the contract. 
Without sufficient contractual or other docu- 
mentation, the OPP risks not being able to clearly 
demonstrate that it is providing policing services to 
municipalities consistently and cost-effectively. 

The costing formula for the OPP's contracts is 
established by a regulation under the Police Ser- 
vices Act and approved by Treasury Board. Direct 
costs represent about 80% of the formula and are 
updated annually based on actual salaries and 
benefits paid to officers, including all increases 
negotiated as part of collective agreements. Other 
support and operating costs make up the rest of the 
formula and are based on provincial average costs 
calculated periodically and applied based on the 
number of officers assigned to the municipality. The 
cost per officer to municipalities for both direct and 
other support and operating costs increased from 
approximately $122,200 in 2007 to $144,000 in 
2011, or 18%. 

Other support and operating costs embedded 
in the formula were not updated annually, so the 
charges did not necessarily reflect current OPP 
costs. Beginning in 2012, all municipalities without 
contracts were billed using the 2010 costing for- 
mula, the most recent approved by Treasury Board. 
However, for municipalities with contracts, the 
OPP is currently billing based on 2003, 2008 and 
2010 formulas, depending on when the five-year 
contract was last negotiated. This is because as the 
contracts expired and were renewed, the most cur- 
rent formula at the time was applied to the contract 
for the entire five-year renewal without provision 
for the OPP to update the formula. Other support 
and operating costs increased from about $18,000 
per officer under the 2003 formula to $29,000 in 
the 2010 formula. As a result, municipalities could 
have experienced increases of up to 60% in these 



costs in the year that the contracts were renewed. 
As of March 31, 2012, the OPP was still billing 32 
municipalities using the 2003 formula and four 
using the 2008. We noted that if all municipalities 
were updated to the current 2010 formula, the OPP 
would have recovered an additional $5.4 million of 
its other support and operating costs annually. 

The OPP has received approval from Treasury 
Board to update and apply the formula on an 
annual basis beginning in 2012. Internal auditors 
also identified that costs in the formula are not con- 
sistently included, updated or validated, and that 
there is no clear documentation for the rationale for 
including or excluding a particular cost. Beginning 
in 2012 with the annual updates to the formula, 
the OPP was to have a formal process in place to 
review all the items used in the costing formula, 
and improve its documentation for determining 
which costs should be included. Better documenta- 
tion should also help improve the OPP's ability to 
disclose its charging practices to municipalities. 

In addition, we found that in 2011, municipal- 
ities covered by 49 of the 118 contracts in place 
received more hours of service than their guaran- 
teed minimums, including three locations that have 
each been over-serviced by more than 10,000 hours 
annually since 2007. However, the OPP is unable to 
recover the additional costs of providing more ser- 
vices than required in the contracts. As well, muni- 
cipalities covered by 21 contracts received fewer 
hours of service than the guaranteed minimum. 
OPP officials told us that these discrepancies could 
stem from officers not properly recording their time 
or because the use of their time had not been prop- 
erly monitored by their supervising officer. 

Annual Credit Reimbursement to 
Municipalities 

OPP detachments operate on an integrated service 
model whereby one detachment may provide poli- 
cing services to several municipalities — contract 
and non-contract — as well as perform provincial 
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policing duties either within or outside of the 
detachment area. 

Officers who work in detachments are not 
designated as specifically municipal or provincial. 
Whatever their duties on a particular day and what- 
ever municipality they work at, officers record their 
activities and location in the OPP's Daily Activity 
Reporting (DAR) system. As part of the billing pro- 
cess, the OPP provides contract and non-contract 
municipalities with a Provincial Service Usage 
(PSU) credit to make up for the time officers in the 
detachment spend performing provincial policing 
duties or working outside of the detachment area 
rather than performing municipal duties. The OPP 
provided an average of $23 million per year over 
the last five years in total PSU credits to municipal- 
ities. These credits were in addition to the police 
services grants managed by the Ministry of Finance. 

For municipalities with contracts, we asked why 
the OPP needed to provide PSU credits of about 
$15.6 million annually when its contracts and 
annual budgets with these municipalities are based 
on hours of service and municipalities are charged 
only for the time spent on municipal policing, thus 
making the credits unnecessary. (For municipalities 
without contracts, annual budgets are calculated 
in a different way that requires providing a PSU 
credit.) In addition, we noted that the PSU credit 
and billing method does not take into account 
whether the minimum required contract hours 
were exceeded by the OPP for municipalities with 
contracts. For instance, five municipalities with 
contracts were each over-serviced by more than 
10,000 hours in 2011 and still received between 3% 
and 8% of their gross costs back in PSU credits at 
the end of year. 

The OPP notes that municipalities rely on PSU 
credits to reduce their police services costs, and 
that any proposal to eliminate the credit would 
need Treasury Board approval, because it would 
place a greater financial burden on municipalities 
at the same time that the updated costing formula 
was being applied to include more of the OPP's true 
costs for providing services. 



Other Provinces 

The charging practices for the OPP's services are 
complex and onerous both on municipalities and on 
the OPP, and, as already noted, not well understood 
by municipalities. In contrast, our understanding of 
the RCMP and Surete du Quebec costing processes 
suggests they appear much easier to administer. For 
example, the RCMP dedicated officers to its munici- 
pal contracts and charged municipalities 80% of the 
cost of those officers. As a result, the RCMP did not 
provide credits to municipalities for any police work 
they performed outside of the municipality, since 
co-operation between police forces is considered 
a normal practice. In Quebec, all municipalities 
with populations up to 50,000 have been required 
since 2001 to use the Surete du Quebec for policing. 
Ten-year contracts are used, and regional muni- 
cipalities are invoiced based on property values 
and population. 

The OPP prepared a briefing note to the Minister 
of Community Safety and Correctional Services in 
2004 proposing to implement a provincial per capita 
or per household methodology for invoicing muni- 
cipalities, but no action was taken. The OPP advised 
us at the time of our audit that it was reconsidering 
a per capita costing methodology and expected to 
present a briefing note to the Ministry in late 2012. 

Assisting Municipal Police Services 

The OPP is sometimes called on to assist municipal 
police forces, but it has never charged municipalities 
for this, even though it has the authority to do so 
under the Act. In some cases, a formal support 
agreement with the OPP is required to allow the 
municipal police force to meet the adequacy stan- 
dards under the Act for a municipality to have its 
own police force. The OPP has calculated that from 
2007 to 2009, the average cost of providing assist- 
ance to other police services was $11.3 million per 
year. However, we were told that this calculation 
included costs that fall under the OPP's provincial 
responsibilities, such as complex cross-jurisdictional 
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investigations, which may not be recoverable, and 
that the OPP's system does not track the strictly 
municipal assistance costs separately. 

As part of its 2012/13 planning, the Ministry has 
recommended to the Treasury Board that it start 
charging municipalities for its assistance. However, 
the OPP noted that it was the smaller municipalities 
that most often required the most assistance with 
major incidents or investigations, and they might 
not be able to afford to pay for such help. 



RECOMMENDATION 7 



To promote better relations with, and consist- 
ent services to, municipalities, and fairer 
and more transparent billing processes, the 
Ontario Provincial Police, in conjunction with 
the Ministry of Community Safety and Cor- 
rectional Services, the Ministry of Finance and 
municipalities, should: 

• seek ways to simplify, and make more 
transparent, its cost-recovery methods 
and consider whether various grants and 
credits should be amalgamated into one all- 
encompassing costing formula; 

• address the issues in its costing and billing 
methods that result in municipalities paying 
different rates and consider phasing in cost 
increases over time rather than when con- 
tracts are renewed; and 

• consider establishing a policy that would 
require identifying all costs for providing 
services to support municipal police forces as 
well as the proportion to be recovered. 

| OPP RESPONSE 

A comprehensive document explaining the 
cost-recovery formula was prepared in August 
2012 in collaboration with the Association of 
Municipalities of Ontario (AMO) and three 
municipal Chief Administrative Officers. This 
document was shared with every municipality 
that receives policing services from the OPP. 



The OPP, Ministry of Community Safety and 
Correctional Services and Ministry of Finance 
have been collaborating on identifying and 
evaluating changes to the current complicated 
system of charges and grants. Proposals for a 
more equitable and simple billing process are 
being developed for consideration as part of 
the government's commitment to economically 
sustainable policing. 

The recovery of costs for providing police 
service to municipalities that receive policing 
from the OPP is governed by the regulations 
under the Police Services Act. The OPP and 
Ministry report annually on costs in accordance 
with these regulations. In addition, the OPP 
brings forward options for additional cost cat- 
egories for inclusion in the cost-recovery model 
if they are found to be directly attributable and 
material to the provision of services. AMO is 
aware of this practice/policy and Police Service 
Boards are also made aware through their 
biannual conferences. 

OPP senior management is open to the con- 
cept of charging municipalities for some services 
that the OPP provides to municipal police forces, 
depending on a variety of factors, particularly 
where core police services are provided that 
would normally be routine business for a force. 



USE OF VEHICLES 

The OPP operates a fleet of about 3,800 vehicles — 
1,250 assigned to officers or civilian staff and the 
rest designated as patrol and pool-usage vehicles. 

In 2008, to comply with Canada Revenue 
Agency regulations, the OPP notified officers and 
civilian staff that they would incur a taxable benefit 
when a vehicle assigned to them was used for 
personal reasons, such as routine commuting from 
home to work. In March 2009, the OPP approved 
a list of job descriptions and positions that were 
deemed exempt from a taxable benefit. 
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The OPP calculates the amount of taxable 
benefit to be reported on employee T4 income slips 
using its fleet bureau system, in which drivers are 
to record their personal mileage each month. Man- 
agers are responsible for reviewing the fleet bureau 
entries at the end of the year to ensure employees 
are properly reporting their mileage. 

In late 2011, the OPP began a review to see if 
it could operate more efficiently by cutting down 
on the number of vehicles it assigned to officers 
and civilian staff and to examine whether taxable 
benefits were being properly administered. The 
review, completed in February 2012, found that 
poor record-keeping and problems with various 
processes resulted in the following: 

• The whereabouts of 200 vehicles listed in the 
fleet management database were not readily 
known, since the vehicles were not listed as 
assigned to a region or investigations bureau. 

• The list of vehicles assigned to staff could not 
be located and had to be recreated. 

• The approved list of job descriptions and pos- 
itions deemed exempt from a taxable benefit 
had not been reviewed and updated since 

it was released in March 2009. The review 
also identified 40 staff listed as exempt, but 
whose current positions did not qualify for the 
exemption. 

• No personal mileage was reported by more 
than half of the 640 non-exempt staff who 
were assigned vehicles. 

OPP staff informed us that, as of September 
2012, they had accounted for all vehicles and issued 
amended T4 tax slips to the 40 employees whose 
positions did not qualify for an exemption. 
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To help adequately manage and control the use 
of vehicles, the Ontario Provincial Police should: 
• improve its record-keeping and other pro- 
cesses for tracking inventory and assigning 
vehicles and capturing personal-use 
mileage; and 



• ensure that its processes result in compliance 
with tax laws that require that any signifi- 
cant personal use of vehicles be reported as 
employee income. 



OPP RESPONSE 



The OPP agrees that the vehicle management 
process implemented in 2008/09 is not meeting 
current needs. A project has been established 
to review the current system and improve the 
management of vehicle taxable benefits. As well, 
all discrepancies between the fleet management 
database and the physical vehicles have now 
been resolved. New processes and policies will 
enable the OPP to effectively and efficiently 
manage vehicle assignments and ensure compli- 
ance with tax reporting requirements. 



DETACHMENT INSPECTIONS 

The Police Services Act stipulates that all police 
forces have quality assurance processes in place for 
the delivery of adequate and effective policing. The 
OPP's Quality Assurance Unit uses two processes for 
conducting audits for this purpose: 

• It provides a Management Inspection Process 
(MIP) questionnaire to each detachment com- 
mander three times a year to assess whether 
the detachment is meeting the standards 

of the Police Services Act and the OPP's own 
policies. 

• It conducts an on-site physical inspection of 
all detachments every two years (three years 
up until mid-2011) to verify the information 
provided in the MIP questionnaire; audit 
high-risk areas such as vaults that hold seized 
firearms, drugs and cash, and storage of the 
detachment's armaments; and examine the 
detachment's financial processes. The Unit 
sends its findings to the detachment com- 
mander and to management at the regional 
headquarters and at General Headquarters. 
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We expected that with these quality assurance 
processes in place, relatively few problems would 
be found at the detachments. However, inspection 
reports that we saw often noted that detachments 
were not complying in a number of areas and were 
told to correct deficiencies. Similarly, we did our 
own inspections at eight detachments and found a 
number of deficiencies, even though, just as with 
inspections by the Unit, the detachments were noti- 
fied in advance of our inspections. The problems 
we found at the detachments included missing 
documentation, the use of improper signatures for 
authorizations, and poor controls over evidence 
and OPP weapons. For example: 

• We noted several cases where physical counts 
of seized items in the vaults did not match 
the inventory listings in the computerized 
Records Management System (RMS) . At one 
detachment, the staff could not provide an 
inventory listing from RMS for seized cash, 
drugs and firearms, and instead were using 

a manually prepared ledger that lacked the 
controls in the RMS system. We found three 
cash court-evidence exhibits in the vault that 
were not recorded in the ledger. At two other 
detachments, the physical count of seized fire- 
arms did not match the RMS inventory listing; 
in one of the cases, the vault contained five 
fewer seized firearms than the RMS listing. 
In several other detachments, the required 
records of items placed in and removed from 
the vaults were not always maintained or the 
items were improperly documented. It was 
also often not clear who had access to court 
evidence in the vaults. 

• Several transfers of seized cash out of detach- 
ments were made without acknowledgment 
from the recipients. At one detachment, 
$7,500 was transferred to another OPP inves- 
tigations bureau without proper signatures, 
and an exhibit of $1,700 in seized cash could 
not be located in the vault. 

• We found cases in which seized items were 
secured in non-approved exhibit storage 



bags, where documentation for items was 
missing, and where only one signature was 
obtained for disposal of illicit drugs when two 
signatures are required. At one detachment a 
civilian staff member was assigned to manage 
the storage vault, which is contrary to policy. 
In some cases, officers' handguns were stored 
in locked firearms lockers that did not have 
the required locking cable or an approved 
lock. As well, we found off-duty officers' pep- 
per spray canisters were not in handgun stor- 
age lockers as required by policy. 
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To ensure that detachments meet legislative and 
policy requirements for ensuring the security 
and integrity of seized cash, drugs and firearms, 
and detachment weapons, the Ontario Provin- 
cial Police should: 

• reassess its quality assurance processes and 
increase senior management oversight of 
results to identify ways to make inspections 
more effective, including the periodic use of 
surprise inspections to promote sustainable 
compliance; and 

• make detachment commanders more 
accountable for ensuring that actions have 
been taken to correct any deficiencies noted. 



OPP RESPONSE 



The OPP has made a strong commitment to 
ensuring that Police Orders and legislative 
requirements for policing standards are consist- 
ently followed. The OPP has investigated and 
resolved the discrepancies identified by the 
Auditor General's review. Through this review, 
the regions have identified accountability, train- 
ing and policy clarity as areas that will improve 
seized asset management. The OPP will review 
the suggestions and implement new processes 
where required. 

The OPP's Quality Assurance (QA) unit 
works in partnership with the regions to 
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continually review and improve QA processes. 
Furthermore, substantial changes were made 
after the Auditor General's recommendations in 
2005 and follow-up report in 2007. 

Currently, the Deputy Commissioner of Field 
Operations and each Regional Commander 
receives a summary of all findings. The QA unit 
works with the detachments and the region to 
ensure all the issues are resolved in a timely fash- 
ion. If an issue is systemic or involves a risk that is 
not resolved, the QA unit will inform the Deputy 
Commissioner. In most cases, errors are related 
to documentation or process discrepancies. 

The OPP will continue to reassess its quality 
assurance processes and ensure detachment 
commanders are accountable. 



EFFECTIVE POLICING 

Community-oriented Policing 

The Police Services Act requires the Minister of Com- 
munity Safety and Correctional Services to develop 
and promote programs for community-oriented 
police services. As well, the Act establishes min- 
imum police service standards and requires every 
chief of police to establish specific procedures and 
processes for community-oriented policing. 

Historically, the OPP's community-oriented 
policing program involved regular meetings and 
communications between a committee of local 
citizens and members of the local detachment, dur- 
ing which the group would deal with local concerns 
about such issues as crime and traffic. The goal of 
the program has been to improve community safety 
and to involve the community in crime prevention. 

In 2010, the Ontario Association of Chiefs 
of Police introduced a new community-policing 
model, which, according to the 2011 OPP business 
plan, was to be the foundation of the force's crime- 
prevention efforts. The business plan indicates that 
the OPP needed to make significant efforts through 
awareness sessions or other communications to 



tell OPP staff about the new model and train them 
accordingly, and to let communities know about it. 

On our visits to eight detachments, we asked 
commanders, sergeants and constables whether, or 
what, they knew about the new community-policing 
model. Some detachment members knew about the 
new model and had even received some training 
or had attended awareness sessions; others were 
unaware of it. We also checked to see whether the 
community-policing committees still existed. In 
three jurisdictions, the committees still met, and in 
another three, the committees existed in some form, 
but had not met recently or were being redeveloped. 
In two detachments, there were no committees. 

In our 1998 and 2005 audits, we reported that 
the community-oriented policing program needed 
to have better management oversight and a pro- 
cess to measure the effectiveness of the program 
against established criteria. We found during our 
current audit that these observations had not been 
addressed. 

Results-Driven Policing 

In 2006, the OPP introduced the Results-Driven 
Policing Accountability Framework for improving 
community safety through targeted crime and 
traffic-enforcement initiatives. Using the framework, 
detachments and regional headquarters collect and 
analyze statistics on rates of certain crimes and the 
numbers of road-related fatalities and injuries, and 
these statistics are discussed at quarterly meetings. 
If, for instance, the statistics show an increase in 
accidents on a certain highway, or a spike in certain 
types of crimes in a particular area, resources may be 
redirected to address the situation. Senior manage- 
ment at General Headquarters reviews the Results- 
Driven Policing reports quarterly and, if necessary, 
requests corrective action. 

This sort of response strategy is useful in that 
it helps detachments focus their resources on 
improving tangible results, but the framework sets 
no targets against which detachments can meas- 
ure their results and compare them with other 
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detachments. Setting targets could allow the OPP 
to more effectively monitor and improve detach- 
ments' performance. 

Response Times 

Communication devices in OPP vehicles have a 
button that officers use specifically to report to 
the communications centre when they arrive at a 
scene, and officers are required to use this button. 
However, unlike municipal police services and 
the RCMP, the OPP does not analyze information 
about how much time elapses between when a call 
is received by one of its regional communications 
centres and when the officer arrives on the scene. 
Consequently, the OPP does not know whether 
average response times are reasonable, particularly 
for more serious or higher-priority calls. OPP 
officials told us it would be impractical to set a 
response-time standard for all calls because of sig- 
nificant geographic differences across the province. 
Nevertheless, keeping track of average response 
times on a detachment-by-detachment basis would 
allow the OPP to monitor officer response times to 
determine whether there were any detachments 
that may need more officers during certain periods. 

The OPP compiled an informal sample of aver- 
age response times from one regional communi- 
cations centre in 2011 and found that an officer 
arrived on the scene within 10 minutes in 78% of 
calls for service and within 40 minutes for 95% of 
calls. No other analyses of response times have been 
carried out, whether of individual detachments, 
regions or the province as a whole. 



RECOMMENDATION 10 



To help ensure that police resources are focused 
on the Ontario Provincial Police's key objectives 
for effective policing, the Ontario Provincial 
Police should: 

• improve the reporting to management on 
the community-oriented policing program 
and the results-driven policing program, 



and establish measures for assessing the 
effectiveness of these programs at individual 
detachments; and 
• monitor average officer response times to 
calls for service for each detachment to 
ensure that adequate response times are 
achieved, particularly for higher-priority 
calls and during peak demand periods. 



PP RESPONSE 



The OPP will develop a reporting mechanism for 
community-oriented policing. The OPP Crime 
Prevention Section has begun communicating 
with regions on how Ontario's Mobilization and 
Engagement Model of Community Policing can 
be used most successfully. The OPP has also 
established a tactical priority-setting process 
enabling officers in the OPP and other police 
agencies to focus on severe crime problems 
affecting communities. 

The Results Driven Policing (RDP) program 
is an accountability process meant to identify 
and respond to issues quickly, rather than com- 
paring benchmarks against other detachments 
and regions. Significant work is done at the 
detachment and regional levels regarding RDP 
statistics and in response to trends and emerging 
issues. Best practices are identified and shared. 
All of those results are in turn rolled-up and 
reviewed monthly at OPP Executive Council 
meetings at General Headquarters. The OPP is 
currently developing a better corporate analysis 
framework and reporting process on the benefits 
and the successes/failures of RDP regional initia- 
tives. In the new year, a streamlined and more 
efficient RDP process will be implemented at the 
detachment, regional and provincial levels. 

The OPP will investigate ways to efficiently 
monitor response times and has set a target to 
provide reports and analysis to senior manage- 
ment by the end of 2013. 
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INFORMATION SYSTEMS 

Officers use two separate OPP systems on a daily 
basis to record their activities: 

• The Records Management System (RMS) 
tracks occurrences, such as those resulting 
from calls for service, and permits analysis 
of case-related information, such as types of 
crime committed, location, people and prop- 
erty involved, witness statements and officers' 
notes. Officers are required to update RMS 
after each occurrence. 

• The Daily Activity Reporting system (DAR) 
is, primarily, a time-accounting system that 
tracks an officer's regular and overtime work 
hours, the number and types of calls for 
service to which an officer responds, and how 
much time each officer spends on activities 
such as traffic patrol, Criminal Code investiga- 
tions and administration. Officers are required 
to update DAR daily. 

Officers, administrators and local and provin- 
cial governments rely on the RMS, which holds 
about 11 million records, to provide accurate data 
quickly. For example, front-line officers use the 
RMS to look up records of known violent offenders, 
vicious animals or high-risk residences, as well as 
to access a suspect's criminal background and other 
vital data. Organizationally, the OPP uses RMS to 
compile crime statistics for policing initiatives such 
as community-oriented and results-driven policing, 
and uses DAR for reporting publicly on the number 
of calls for service to which it responds. 

The OPP has acknowledged concerns about the 
quality of data in RMS. Random internal audits 
have revealed a number of data-quality issues, 
including the fact that occurrence information is 
often incompletely or poorly documented, and that 
occurrences that are already recorded in the system 
do not show up during RMS searches by officers. 

Data integrity problems are more prevalent in 
DAR. In 2005, we recommended that the OPP insti- 
tute procedures to require supervisory officers to 
oversee the completeness and accuracy of the infor- 
mation entered into the system by officers. This 



was only partially implemented, and a supervisor's 
approval is required only when officers record over- 
time hours. Supervisors are also required to ensure 
weekly that officers have updated DAR, but they 
do not check or approve the officers' data entries. 
By comparison, the Surete du Quebec requires that 
supervisors approve officers' activity reports on a 
daily basis. 

There is data overlap between the occurrences 
recorded in RMS and the calls for service recorded 
in DAR, with between 50% and 80% of similar data 
in both. Since there is no linkage between the two 
systems, data has to be manually entered twice. 
The OPP does not have any processes that regularly 
compare or reconcile the information in RMS with 
that in DAR. Our own analysis comparing the similar 
data in the two systems for 2011 noted a significant 
difference between the 635,000 calls for service in 
DAR and the approximately 815,000 occurrences 
reported in RMS. For example, some of the differ- 
ences in the number of occurrences or calls for ser- 
vice between RMS and DAR were approximately as 
follows: 4,000 thefts, 1,200 assaults, 13,000 alarm 
responses, 26,000 police assistances, 130,000 traffic 
incidents and 6,000 incidents relating to compulsory 
auto insurance. The general view among the officers 
we spoke to at the OPP is that the occurrences data 
from RMS are more reliable than the calls-for-service 
data in DAR; however, it is the DAR data that is 
published in annual reports and used for staffing 
deployment models. 



RECOMMENDATION 11 



To help ensure that its two key information sys- 
tems contain accurate information that can be 
reliably used for managing and reporting on its 
policing activities and on crime and traffic occur- 
rences, the Ontario Provincial Police should: 
• assess the extent to which the Records 
Management System and Daily Activity 
Reporting systems do not reconcile with 
each other for critical data such as occur- 
rences and calls for services; 
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• consider whether periodic supervisory 
approval of officers' daily or weekly data 
input would help minimize inconsistent and 
inaccurate data between the two systems; and 

• on a longer-term basis, assess the cost/ 
benefit of system changes that would enable 
officers to enter information such as occur- 
rences and calls for service only once to 
update both systems. 

| OPP RESPONSE 

As the Auditor General recommends, the OPP 
has initiated work to identify where RMS and 
DAR data do not reconcile. Particular attention is 
being paid to the areas of occurrences and calls 
for services. Simplification of system input and 
the elimination of duplication is a specific deliv- 
erable of a project established to review the DAR 
system. The OPP is exploring opportunities for a 
customizable off-the-shelf product developed by 
the private sector, and estimates that a product 
will be procured within the next 18 months. 

In addition, the OPP will develop policy and 
procedure around regular review and signoff 
for officer-entered data. The expansion of civil- 
ian data entry into RMS, which is to begin in 
2013/14, is expected to result in improved data 
entry because of direct supervision and oversight. 



PERFORMANCE MEASUREMENT 
AND REPORTING 

The OPP uses its annual report to publicly give 
details about its programs, services and activities; 
its three-year strategic plan; motor-vehicle fatalities 
and injuries; and crime levels and clearance rates. 
Crime statistics are reported in the same uniform 
reporting format used by police forces across 
Canada. As of August, 2012, the most recent OPP 
annual report covered the 2010 calendar year. 

While the OPP reports extensive information 
on its policing activities, crime and motor-vehicle 



accidents, it does not report on its own efficiency. 
By comparison, municipalities and their police 
forces typically report on their operating costs for 
police services on a per capita or per household 
basis. Other police forces set targets for achieving 
efficiency levels, such as times for its communica- 
tions centre to respond to an emergency call and 
officer time lost to illness. In addition, the OPP does 
not include in its annual report the percentage of 
time its officers spent on administrative duties. 

In its 2011-13 Strategic Plan, the OPP identified 
four goals: 

• public safety, with emphasis on excellence in 
delivering core police services; 

• strong, effective partnerships with commun- 
ities, stakeholders and colleagues; 

• a sustainable workforce that demonstrates 
expertise, pride and dedication; and 

• efficiency and effectiveness in an increas- 
ingly complex and challenging policing 
environment. 

The force established performance indicators 
for each goal, but very few are measurable based 
on actual performance, and no targets have been 
set. For the efficiency and effectiveness goal, for 
instance, the performance indicators include having 
"uniform workloads statistics," "offering financial 
training opportunities," and "response to various 
inquests, inquiries and audits." In its 2010 annual 
report, the OPP reported on results pertaining to 
effectiveness achieved on its 2008-10 Strategic 
Plan, including a list of activities completed, but 
no measurable results were provided on the cost- 
effectiveness of its operations. 

The OPP conducts a community satisfaction 
survey every two years. In its last annual report, 
the force said the 2009 survey found that 95.4% of 
Ontarians felt safe or very safe in their communities. 
The results were based on interviews with the gen- 
eral public in specific communities about their views 
of the OPP. However there was no way to ascertain 
whether respondents actually had any contact with 
the OPP. We noted that the RCMP annually reports 
on the percentage of clients it dealt with who are 



Ontario Provincial Police 



satisfied that the force is providing high-quality ser- 
vices. In 2010/11, the RCMP's target was 80% and 
the actual performance was 79%. 



RECOMMENDATION 12 



While the Ontario Provincial Police provides 
good information on crime rates and its activ- 
ities and services, additional information to 
enable the public to assess its cost effectiveness 
and operational efficiency is needed. It should 
also periodically and independently survey com- 
munity residents who have had recent contact 
with the force to determine their satisfaction 
with the service they received. 



and operational efficiency within police agen- 
cies. The OPP will ensure that the targets it sets 
for its goals are measurable. 

The annual OPP Community Survey includes 
questions on effectiveness as well as questions 
specific to residents who have had recent contact 
with the force. The results are analyzed inter- 
nally. The OPP agrees with the Auditor General's 
recommendations on reporting the results and 
will post its 2012 survey results on the public 
Internet site. Also, detachment survey results are 
available to Detachment Commanders to share 
with their communities. 



OPP RESPONSE 

The OPP will research best practices with regard 
to measuring and reporting on cost-effectiveness 



Chapter 3 
Section 
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Tax Collection 




Taxes are the province's largest source of revenue. 
The Ontario Ministry of Finance (Ministry), 
through its Collections Branch (Branch), is 
responsible for collecting a significant portion 
of the unpaid taxes owed to the province. The 
Branch's head office is in Oshawa, with regional 
and district offices in Toronto, London and Ottawa. 
The Branch's collections process involves initially 
sending notices by mail, contacting the taxpayer by 
phone, and sometimes visiting in person. If these 
actions fail to recover amounts owing, collectors 
have other means at their disposal, such as garnish- 
ments or registering liens and warrants for seizure 
and sale of the taxpayer's property. 

Accounts are prioritized on the basis of risk, 
using criteria such as the amount owing, number of 
times or length of time in collections, whether there 
is a history of broken promises, and if any legal 
action has already been taken. Once their priority 
has been set, the accounts are assigned according to 
collectors' seniority and experience. 

As of March 31, 2012, approximately 90% of the 
taxes owing that the Branch was responsible for col- 
lecting related to Corporations Tax and Retail Sales 
Tax. The Canada Revenue Agency (CRA), which is 
responsible for collecting personal income tax on 



behalf of the province, also began administering 
Corporations Tax on behalf of the province in Janu- 
ary 2009. Similarly, in July 2010 the Harmonized 
Sales Tax, also administered by the CRA, replaced 
the provincial Retail Sales Tax. As a result, approxi- 
mately 75% of the Branch's staff of almost 400 were 
transferred to the CRA effective March 2012. How- 
ever, the Ministry remains responsible for collecting 
Corporations Tax and Retail Sales Tax amounts 
owing prior to the transfer of the administration of 
these taxes to the CRA. The Ministry expects that by 
2014 it will have wound down most of its work on 
collecting these amounts. 

In the 2011 Ontario Budget, the government 
proposed centralizing the collection of all govern- 
ment non-tax revenue within the Ministry of 
Finance. Under this proposal, the Ministry's Collec- 
tions Branch would continue to collect taxes that 
it administers, but would also become responsible 
for collecting non-tax revenue on behalf of other 
provincial ministries. At the time of our audit, the 
Ministry of Agriculture, Food and Rural Affairs 
was the first ministry scheduled to transfer non-tax 
receivables — in this case, overpayments to crop 
producers totalling approximately $29 million — to 
the Ministry of Finance in fall 2012. 



Tax Collection 



Audit Objective and Scope 



Our audit objective was to assess whether the Min- 
istry of Finance (Ministry), through its Collections 
Branch (Branch), had adequate systems, policies 
and procedures in place to: 

• maximize the recovery of taxes owed to the 
province; and 

• reliably measure and report on the effective- 
ness of these collection efforts. 

Ministry senior management reviewed and 
agreed to our objective and associated criteria. 

The Branch's head office in Oshawa is respon- 
sible for accounts representing about 80% of the 
taxes that the Branch is responsible for collecting. 
Our audit, therefore, focused on a review and 
analysis of relevant collection accounts at this 
office, its administrative policies and procedures, 
as well as discussions with appropriate staff. 
However, we also reviewed a number of accounts 
assigned to collectors in the Branch's regional and 
district offices. 

The scope and extent of our audit work also 
took into account several recent initiatives by the 
Ministry. The Branch hired a consulting firm in 
2009 to review its operations and recommend best 
practices and potential collection tools from juris- 
dictions similar to Ontario. In 2009, the Branch also 
co-founded the Inter- Jurisdictional Tax Operations 
Network (ITON), whose membership includes 11 
American states and seven Canadian provinces, to 
share knowledge about the collections process. As 
part of the audit, we reviewed and assessed best 
practices identified through ITON surveys and the 
external review commissioned by the Branch. In 
August 2010, the Ministry's Internal Audit Division 
conducted a review of the collection processes 
of accounts deemed to be low-risk. This division 
also annually reviews the Ministry's allowance for 
doubtful accounts and the write-off process. And 
in 2011, in anticipation of an expanded role for its 
Collections Branch, the Ministry commissioned an 



accounting firm to assess and identify the structure 
of a centralized collection service as proposed in 
the 2011 budget. 

We also reviewed recent reports on effective 
collection practices from associations that represent 
private collection agencies in North America. 



Summa 




Over the last five years, the province has generated 
about $330 billion in taxation revenue, of which the 
Ministry of Finance's Collections Branch (Branch) 
collected about $6 billion, with the majority of 
the remaining amount being remitted voluntarily. 
Although some write-offs are to be expected in any 
collection process, the Branch expects that it may 
need to write off up to $1.4 billion of the $2.46 bil- 
lion in taxes owing to the province that the Branch 
was responsible for collecting as of March 31, 
2012. The $1.4-billion amount is predominantly 
made up of older accounts that have accumulated 
over a number of years, and it has been previously 
expensed in the government's financial statements. 

In recent years, the Branch has taken some 
initiatives to strengthen its collections process. In 
2008, collection activity for all tax statutes except 
Corporations Tax and Self-employed Health Tax 
migrated to a new system called OntTax. With 
the implementation of OntTax, the Branch has 
improved how it prioritizes and assigns accounts to 
collectors. The Branch has also recently developed 
guidelines to assist collectors in carrying out and 
documenting collections activities, as well as imple- 
menting a tool that analyzes an account's collection 
history to predict the likelihood of payment. 

To understand why the Branch needed to write 
off a significant amount of taxes owing, we exam- 
ined the collection process both for active accounts 
and for those that the Branch was considering writ- 
ing off. We found that in most cases we reviewed, 
the collection actions taken were not timely and 
the enforcement tools available were not utilized 
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fully. Some of our more significant observations are 
as follows: 

• Taking prompt action is vital in collecting 
debts. Research shows that the probability 
of full collection on a delinquent account 
drops dramatically as time passes. Our work 
indicated that once an account entered collec- 
tions, it took an average of seven months for 
collectors to attempt to reach the taxpayer by 
phone. We also noted that in more than two- 
thirds of the cases in our sample, there was at 
least one instance where no collection action 
was taken for six months or more. 

• Visiting the taxpayer's premises will often 
increase the likelihood of collecting the 
amount owing. Field visits were warranted 
but were not made in a number of accounts 
that we reviewed. For example, the Branch 
tried unsuccessfully for nearly two years 

to reach by phone a taxpayer who owed 
$100,000 in Retail Sales Tax and had broken 
a payment arrangement. Making prompt field 
visits to the taxpayer's retail or business loca- 
tion would be a more effective strategy, espe- 
cially when significant amounts are involved. 

• The Branch had in most cases registered liens 
and warrants on properties appropriately 
when such action was considered necessary. 
However, in a number of the cases that we 
reviewed, the Branch did not enforce warrants 
for the seizure and sale of property because 

of concerns for the viability of the taxpayer's 
business and the impact on the taxpayer's abil- 
ity to pay outstanding amounts. Even when 
businesses had ceased to operate, we noted 
cases where the Branch still failed to enforce 
existing warrants to determine whether any 
assets remained to help pay the debt. 

• The Branch may arrange interim payment 
plans if the taxpayer has outstanding returns 
to file or needs time to determine a permanent 
payment arrangement. Payment arrange- 
ments were in place in nearly half of the 
accounts that we reviewed. However, contrary 



to Branch guidelines, multiple interim pay- 
ment arrangements that covered only a small 
portion of the total debt had been in place for 
extended periods in many of these accounts. 
• The Branch did not always make full use of 
partnership and information-sharing agree- 
ments with third parties. For example, the 
Branch may seek to have a delinquent tax- 
payer's motor vehicle dealer or liquor licence 
suspended or revoked. We noted cases where 
the Branch did not request such action on a 
timely basis, or at all, after normal collection 
efforts had been exhausted. 
Corporations Tax and Retail Sales Tax make 
up 90% of the Branch's tax-receivable inventory. 
Responsibility for administering these taxes was 
recently transferred to the Canada Revenue Agency 
(CRA), but responsibility for collecting the amounts 
owed to the province pre-transfer remains with 
the Branch. As a result of the transfer, the Branch 
recently lost three-quarters of its workforce, includ- 
ing managers, collectors and support staff. Of the 
remainder, more than one-third is management and 
support staff, and the caseloads of collectors have 
in many cases doubled and in some cases tripled. 
This may result in even more write-offs than cur- 
rently expected. The Branch received approval in 
the 2009/10 fiscal year to hire temporary employ- 
ees to compensate for the loss of its personnel to the 
CRA. At the time of our audit, the Branch had not 
fully evaluated its post-transfer staffing needs and, 
as a result, no additional staff had been brought 
on board. 

In order to oversee collection activities effect- 
ively, managers should have access to sufficient and 
timely operational and performance information. 
However, we found that reports produced by the 
Ministry's information systems did not adequately 
support the oversight of the collection function. The 
Branch's performance measures are also not suf- 
ficient to properly evaluate collection efforts at the 
branch level and at the individual collector level. 



Tax Collection 



Detailed Audit Observations 



The Ministry appreciates the recommendations 
made by the Auditor General to improve the col- 
lections function as we move toward the goal of 
centralized collections of tax and non-tax debt, 
as proposed in the 2011 Ontario Budget. 

Of the $1.4 billion in write-offs referenced 
in the Auditor General's report, $600 million 
has been approved for write-off in 2012. Of 
the remaining $800 million, approximately 
$400 million is in bankruptcy/insolvency pro- 
ceedings subject to federal legislation and from 
which Ontario may receive some future pro- 
ceeds; $200 million is still in active collection; 
and the remaining $200 million is slated for 
write-off because progressive collection action 
has been exhausted. Furthermore, prior to 
writing off accounts, the Branch registers legal 
actions (personal property liens and warrants 
of seizure and sale) to secure the Crown's inter- 
est should any undisclosed assets be identified. 
Historically, this has resulted in the recovery of 
monies from written-off accounts. 

In recent years, the Collections Branch has 
worked to enhance its processes, tools and best 
practices to improve the collections process. For 
example, the tax administration system, Ont- 
Tax, implemented in November 2008, provides 
the functionality for an automated monthly 
statement for all accounts with a balance, along 
with automated letters detailing specific actions 
on the account to encourage debtors to make 
contact with the Branch and resolve their tax 
obligations. The Branch has also introduced 
data analytics and performance measures, and 
initiated and led regular meetings of an inter- 
jurisdictional group of North American collec- 
tion administrators to identify and leverage best 
practices specific to collections operations. 



The province generated approximately $330 bil- 
lion in taxation revenue over the last five years, of 
which the Ministry of Finance's Collections Branch 
(Branch) collected about $6 billion, with most of 
the remaining amount being remitted voluntarily. 

As of March 31, 2012, the total amount of the 
taxes owing that the Branch was responsible for 
recovering was $2.46 billion. However, of this total, 
about $680 million has been deemed uncollect- 
ible and therefore is to be written off. Staff are 
focusing their collection efforts on the remaining 
$1.78 billion in active accounts, of which the Branch 
estimates that another $720 million may need to be 
written off. 

Corporations Tax and Retail Sales Tax make 
up most of the active amounts owing, as shown in 
Figure 1. Because administration of these taxes has 
been transferred to the Canada Revenue Agency, 
the Branch's current priority is to wind down the 
existing pre-transfer inventory of accounts by 2015. 

Figure 1: Active Amounts Owing by Tax Statute as of 
March 31,2012 

Source of data: Ministry of Finance 




— Retail Sales Tax, $641 (36%) 
Employer Health Tax 1 , $71 (4%) 
Others $107 (6%) 



Corporations Tax, $964 (54%) 



1. Includes Self-employed Health Tax (SEHT). 

2. Includes Motor Fuels and Tobacco Tax, Land Transfer Tax, Mining Tax, Beer 
and Wine Tax. 
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Figure 2: Collections Process 

Source of data: Ministry of Finance 



Account enters collections when payment is in default 



System-generated collection letter is sent 



Attempts to contact taxpayer may be made 



Final notice is sent 



Personal property (e.g. vehicles, furniture, equipment) liens 
and warrants may be registered for future seizure and sale 



Real property (e.g., land and buildings) liens 
may be registered and garnishments may be issued 
v . y 



Liquor or motor vehicle dealership licences 
may be suspended or revoked where applicable 



Warrants for seizure and sale of properties may be enforced 



In cases where a corporation has insufficient 
assets or has filed for bankruptcy, the directors 
of a corporation may be held liable for amounts owing 



Account can be recommended for write-off 



Note: 

At any point in the process, a collector is encouraged to attempt to negotiate a 
payment arrangement, and can: 

• obtain security, e.g., in the form of a bank guarantee or a letter of credit; 

• conduct a field visit to, e.g., verify taxpayer information, review books and 
records, or inspect assets; or 

• refer trie account for an audit. If fraud is suspected, trie account may be 
referred for special investigation. 



For the most part, it will assess the need to write 
off accounts that are old and where the likelihood 
of collecting is low, and then focus efforts on the 
remaining accounts. 

COLLECTIONS PROCESS 

Overview 

The flowchart in Figure 2 illustrates the process 
that the Branch typically follows in collecting 
amounts owed. 

The timeliness of collection actions is the most 
important factor in successful recovery of amounts 
owing. A recent survey of its members conducted by 
an association that represents commercial collect- 
ors responsible for 80% of the claims in the United 
States found that the probability of full collection 
on a delinquent account drops dramatically as time 
passes. Its survey results indicated that after only 
three months, the probability of collecting on an 
account drops to less than 70%; after six months, 
collectibility drops to about 50%; and after a year, it 
drops to a little over 20%. After two years, the prob- 
ability of full collection on a delinquent account 
is less than 10%. The association noted that other 
keys to successful collections include keeping credit 
records, such as addresses and contact information, 
current; periodically reviewing collection policies 
and procedures, as well as rigid adherence to these 
policies; and ensuring that commitments of a final 
payment accompany all partial payments. 

As shown in Figure 3, as of March 31, 2012, only 
about $390 million or 22% of the active amounts 
were less than a year old, while approximately 
$925 million or more than half were two years old 
or more. Due to the lower success rate in collecting 
older accounts, the Branch estimates that $720 mil- 
lion (mainly consisting of Retail Sales Tax and 
Corporations Tax) of these active amounts owing 
will also need to be written off. Combined with the 
more than $680 million already scheduled to be 
written off, this means that the Branch will need 
to write off $1.4 billion of the $2.46 billion in taxes 
owing to the province as of March 31, 2012. 



Tax Collection 



Figure 3: Age of Active Accounts Receivable as of 
March 31, 2012 

Source of data: Ministry of Finance 





Active A/ R 




Age 


($ million) 


% 


Less than 1 year 


391 


22 


1 year to less than 2 years 


361 


20 


2 years to less than 4 years 


343 


19 


4 years to less than 10 years 


451 


25 


10 years and older 


131 


8 


Subtotal 


1,677 




Other* 


107 


6 


Totals 


1,784 


100 



* Includes accounts whose age could not be determined. 



Collection Activities 

The Branch has taken steps in recent years to better 
guide collectors through the collections process by, 
for example, developing procedures for activities 
such as carrying out searches and registering liens 
and warrants on properties. The Branch has also 
established standards to ensure that pertinent 
information from collection activities is docu- 
mented fully and properly. 

To understand why the Branch's collection 
efforts have not been successful in recovering a 
significant amount of taxes owing, we examined 
the collection process through a review of the active 
accounts as well as accounts that the branch has 
deemed uncollectible. In most cases, collection 
actions should have been initiated sooner and the 
available enforcement tools used more effectively. 
• The sooner that collectors establish contact 
with a delinquent taxpayer, the greater the 
chance of collecting on that liability. In the 
accounts that we reviewed, after the system 
sent collection notices automatically, it took 
an average of seven months for collectors to 
attempt to reach the taxpayer by phone. In 
several cases this initial contact was made 
two to five years after the account entered 
collections. We also noted that after the 



initial contact, in more than two-thirds of 
the accounts that we reviewed there was at 
least one instance where the account was not 
touched for six months or more. The average 
length of time where there was no activity on 
the account in these cases was 11 months. 

• Visiting the taxpayer's premises is often neces- 
sary to verify information about the client 
and the state of the business and, in some 
instances, to determine whether there might 
be assets that could be used to help recover 
amounts owing. The 2009 external review of 
Branch operations noted that field visits early 
in the collection process add the most value 
when businesses are still in operation and 
assets remain in place, but that staff tend to 
exhaust all options from the desk before pro- 
gressing to a field visit. Indeed, in a number 
of cases that we reviewed we felt that field 
visits were warranted but were not under- 
taken. For instance, in one case the Branch 
negotiated a payment arrangement of $2,400 
a month in October 2009 with a client who 
owed $100,000 in Retail Sales Tax. However, 
after paying only a little more than $5,300, 
the client made no further payments. Over 
the next 22 months, the Branch attempted to 
reach the taxpayer by phone without much 
success. An immediate field visit would have 
been a more effective strategy for attempting 
to establish contact with the taxpayer sooner. 
The taxpayer finally contacted the Branch in 
February 2012 only because the renewal of 
the establishment's liquor licence was in jeop- 
ardy due to the debt. At the time of our audit, 
this amount was still outstanding. 

• The Branch may take legal actions such as 
registering liens on properties and warrants 
for their future seizure and sale if the collector 
is unable to contact the taxpayer or identifies 
a risk of loss. The Branch may also take these 
legal actions when a payment arrangement 
cannot be negotiated with the taxpayer or an 
existing agreement is broken. For the accounts 
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that we reviewed, we noted that the Branch 
had in most cases registered liens and war- 
rants on properties appropriately when such 
action was considered necessary. However, we 
noted in a number of the cases we reviewed 
that the Branch had not enforced warrants for 
seizure and sale of properties, for fear of put- 
ting the taxpayer out of business and thereby 
reducing any opportunity to collect amounts 
owing. While we agree that collectors must 
exercise judgment in enforcing warrants, we 
noted that in several instances businesses had 
ceased to operate but the Branch still failed 
to enforce existing warrants. For example, 
in one case a restaurant owner broke a pay- 
ment arrangement after paying only 5% of 
a $522,000 tax debt. After several attempts 
in 2011 to contact the taxpayer, the Branch 
learned that the company had closed its bank 
accounts. While the Branch did revoke the 
restaurant's liquor licence, it did not enforce 
warrants that had already been registered for 
the seizure and sale of property even though 
there may well have been assets available to 
help pay the debt. 

Directors of companies can be held liable for 
certain tax amounts owing if previously regis- 
tered warrants are enforced and no assets are 
found, or if the company declares bankruptcy 
and the Branch files a proof of claim against 
its assets within six months. In several cases, 
we noted that the Branch could not pursue 
this option because it had not enforced the 
warrants it had registered on properties. One 
company that owed $384,000 in taxes filed 
for bankruptcy, but the Branch failed to file 
a proof of claim in time and therefore lost 
any opportunity to recover taxes owing by 
this method. 

In certain cases where the taxpayer is will- 
ing to pay amounts owing but cannot do so 
immediately or in full, the Branch may negoti- 
ate an agreement with the taxpayer for the 
payment of the amount owing over a period 



of time. Interim payment plans can also be 
arranged if a taxpayer has outstanding returns 
to file or needs time to determine a permanent 
arrangement. According to Branch guidelines, 
interim payment plans are only a short-term 
measure and do not satisfy the taxpayer's 
outstanding liability to the province. Payment 
arrangements were in place in nearly half 
of the accounts that we reviewed. However, 
multiple interim payment arrangements that 
covered only a small portion of the total debt 
had been in place for extended periods, which 
was contrary to Branch guidelines. According 
to the 2009 external review of the Branch, 
some U.S. jurisdictions use automated debit 
payments, which have been found to signifi- 
cantly reduce the default rate on payment 
arrangements. At the time of our audit, the 
Branch had not formally considered adopting 
this practice. 

Use of Third-party Information 

The use of third-party information to corroborate 
taxpayer information, locate an individual or busi- 
ness, or obtain other data is a critical component 
for successful collection. 

Once an account enters collections, the Branch 
has access to the Ontario Business Information 
System maintained by the Ministry of Government 
Services, to the Ministry of Transportation's driver 
and vehicle licensing and registration databases, 
and to the Municipal Property Assessment Corpora- 
tion's database. Although we found that the Branch 
uses these databases to search for individuals and 
businesses and any related assets, it did not always 
commence the searches on a timely basis. In our 
review of a selection of accounts, the average 
length of time between an account entering collec- 
tions and the first search was 12 months. 

For a number of years, the Branch has had 
partnership agreements with some provincial 
agencies and the Canada Revenue Agency (CRA) 
to assist its collection effort, as shown in Figure 4. 
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Figure 4: Partnership Agreements for Assisting the Collection Effort 

Source of data: Ministry of Finance 



1 Agency 


Year Effective 


Purpose of Agreement 


Alcohol and Gaming 
Commission of Ontario (AGCO) 


2003 


Ensures that applicants will not be granted a liquor licence renewal or be 
able to transfer an existing licence until they resolve any tax arrears or tax 
returns in default. 


Ontario Motor Vehicle Industry 
Council (OMVIC) 


1996 


Allows the Ministry to recommend that a motor vehicle dealer's right to sell 
in Ontario be revoked if they fail to resolve any tax arrears. 


Canada Revenue Agency (CRA) 


1982 


Revenue sharing: Enables sharing of proceeds where both the CRA and the 
Ministry have taken collection action against a common debtor. 




1998 


Revenue set-off: Allows for set-off of revenue from the CRA to recover debts 
owed to the Ministry, and vice versa. 



However, we found the Branch did not make full 
use of the avenues available through its partnership 
agreements. In several cases where taxpayers held 
a vehicle dealer registration issued by the Ontario 
Motor Vehicle Industry Council (OMVIC) or a 
liquor licence issued by the Alcohol and Gaming 
Commission of Ontario (AGCO), the Branch did 
not initiate having the licences revoked on a timely 
basis, or at all, after normal collection efforts had 
been exhausted. For example, the Branch warned 
a liquor licensee who owed $1.1 million in Retail 
Sales Tax that it would apply to the AGCO to have 
the licence suspended or revoked if the taxpayer 
did not contact the Branch within 30 days. For 
approximately six months after receiving the 
warning letter, the taxpayer did not respond, yet 
the Branch did not begin proceedings to suspend 
or revoke the liquor licence. Subsequently, after 
repeated attempts by the Branch over a one-and-a- 
half -year period, the taxpayer agreed to a payment 
arrangement. However, the taxpayer defaulted 
after making only three payments totalling approxi- 
mately 5% of the debt outstanding. Again at this 
point, the Branch did not take any action to have 
the liquor licence suspended or revoked. Two years 
after breaking the payment arrangement, the tax- 
payer voluntarily surrendered the liquor licence just 
as it was about to expire. At the time of our audit, 
this amount was still outstanding. 

The Branch also needs to make better use of 
information that the CRA may have on delinquent 



taxpayers. Such information could be used to cor- 
roborate details that the Branch has on file or to 
identify taxpayers' other sources of income. The 
Branch acknowledged that the CRA exchange of 
information is a valuable tool that is available for 
use during the collection process, but that it is not 
used routinely. 

Out-of-province Accounts 

At the time of our audit, a Branch analysis revealed 
that more than $320 million in taxes was owed by 
individuals and businesses whose mailing address 
was outside Ontario. According to the Branch, 
while it can follow up with these debtors by send- 
ing notices or by telephoning, it cannot initiate 
any legal actions such as garnishing bank accounts 
or registering liens and warrants on properties 
if the accounts and properties are also out of the 
province. The Branch indicated that the current 
legislative framework may not allow the Ministry 
or a private collection agency that may be acting 
on its behalf to enforce debt outside the province. 
It is exploring options including requesting legisla- 
tive changes that would permit it to sign inter- 
provincial agreements to allow it to initiate more 
effective collection activities to recover its tax debts 
outside Ontario. 
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RECOMMENDATION 1 



To maximize the recovery of amounts 
owing, the Ministry of Finance's Collections 
Branch should: 

• make initial contact with delinquent taxpay- 
ers sooner and carry out follow-up efforts, 
including field visits, in a more continuous 
and timely manner; 

• make better use of all available collection 
and enforcement tools, including partner- 
ship and information-sharing agreements 
with other parties; and 

• continue to consider options, including 
obtaining any legislative authority that may 
be needed to allow it to initiate legal actions 
to collect debts from businesses and individ- 
uals residing outside the province. 

| MINISTRY RESPONSE 

The Branch will continue to enhance and 
strengthen its collection processes — for 
example, by developing best practices to guide 
collectors in carrying out and documenting 
collection activities executed or not executed, 
and by improving timeliness of actions and how 
work is prioritized based on risk. The Branch 
will take into consideration the right balance 
of initiating actions based on third-party agree- 
ments that result in compliance, and actions 
that limit a business's ability to remain oper- 
ational. The Branch will also initiate a review 
of its payment arrangements in the fall of 2012, 
and recommendations based on the results will 
be implemented by March 31, 2013. 

The Branch is continuing to dialogue with 
the CRA to enhance our information-sharing 
ability. Process improvements from this dialogue 
will be implemented immediately. The Branch 
is also working to expand the tender contract 
compliance program to the broader public sec- 
tor to help ensure that grants are not paid out to 
businesses that owe a tax debt to the province. 



It is anticipated that expansion of this program 
will begin in March 2013. 

Ontario legislation is currently not enforce- 
able in other jurisdictions. The Branch will 
continue to work with legal services and inter- 
jurisdictional partners to explore options for col- 
lecting debt from taxpayers outside Ontario. This 
issue is to be discussed at the Inter-Provincial 
Territorial Tax Conference in September 2012. 
Viable options will be proposed for consideration 
by decision-makers. 



Staffing 

On July 1, 2010, the provincial Retail Sales Tax 
(RST) was replaced by the Harmonized Sales Tax 
(HST), which the CRA administers. As part of the 
amalgamation and transfer of the administration 
of RST, the Branch in March 2010 negotiated an 
agreement with the CRA in which all Ministry 
employees affected by the wind-down of the prov- 
incial RST received job offers from the CRA. As a 
result of this agreement, in March 2012 the Branch 
lost three-quarters of its staff, including managers, 
collectors and support personnel. Prior to the trans- 
fer, the Branch had a workforce of 389 people — 264 
collection and insolvency officers and 125 manage- 
ment and support staff. 

Since the transfer, only 62 collection and 
insolvency officers remain and are now responsible 
for the core collection function. Management and 
support staff number 34 and make up more than 
one-third of the Branch's workforce. Account case- 
loads have in many cases doubled for most levels 
of collection officers since the transfer, and in some 
cases they have tripled. Such a significant reduction 
in the Branch's workforce, especially at the collector 
level, and the consequent increase in caseloads will 
further hinder the effectiveness of collection efforts 
on accounts where the probability of recovery still 
exists, and may well result in even more write-offs 
than expected at present. 
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Prior to March 2012, the Branch had a dedicated 
Field Services Unit of eight field officers who sup- 
ported the desk collectors by visiting businesses 
and residences to review clients' records, inspect 
and appraise assets, and in some cases negotiate 
payment arrangements. The field officers were 
also responsible for attending AGCO and OMVIC 
hearings for the possible suspension of licences 
when taxes were in arrears or returns were not 
filed. This unit was eliminated with the transfer 
of staff to the CRA; the field functions are now 
handled by senior RST and Corporations Tax (CT) 
collectors whose caseloads have in many instances 
doubled, as indicated above. This combination 
of significantly higher caseloads and additional 
responsibilities may result in even fewer site visits 
and additional delays in the desk collection func- 
tion, further decreasing the likelihood of collecting 
amounts owing. 

The Branch received approval in the 2009/10 
fiscal year to hire up to 74 temporary staff up to 
March 2014. It had hired some temporary staff to 
work until March 2012 to compensate for its loss of 
staff as a result of the transfer of the administration 
of CT to the CRA. However, at the time of our audit, 
the Branch had not fully evaluated its needs related 
to the amalgamation and transfer of the administra- 
tion of RST and, as a result, no additional staff had 
been brought on board. The Branch informed us in 
August 2012 that this analysis had been undertaken 
and actions such as hiring temporary staff were 
under way. 

While other Ontario ministries use private 
agencies to collect non-tax debt, the Ministry of 
Finance's Collections Branch does not outsource 
any part of its collection function. The 2009 exter- 
nal review of the Branch's operations suggested 
that the Ministry evaluate the use of private agen- 
cies for collections. The review scanned practices 
in seven jurisdictions that had a similar mandate to 
that of Ontario and found that all used third-party 
agencies to some extent in their collection efforts. 
Typically, these jurisdictions used private collection 
agencies for low-value accounts or accounts where 



all other means of collections had failed. Similarly, 
a study conducted by the Inter- Jurisdictional Tax 
Operations Network, co-founded by the Branch, 
found that six of nine North American jurisdictions 
that were surveyed had considered outsourcing 
collections on a limited basis as a way to share the 
workload and increase efficiency — for example, 
on low-value accounts, accounts located outside 
the jurisdiction, and accounts where internal col- 
lection had yielded minimal results. Some of these 
jurisdictions have had some success in using private 
collection agencies on tax debts, but have required 
legislative authority identifying the information 
that the jurisdiction could give the agencies prior 
to outsourcing. 

We should also note that the Branch still has to 
determine the resources needed to fully analyze 
outstanding defaults — that is, corporations that 
have not filed returns. Specifically, as of March 31, 
2012, there were more than 200,000 CT accounts 
with approximately 500,000 defaults. At the time 
of our audit, the Branch had not yet analyzed 
these accounts to identify the age of the defaults 
or the filing history of the corporations to deter- 
mine the likelihood of these defaults resulting in 
taxes payable. 



RECOMMENDATION 2 



To mitigate the impact of the significant loss of 
its staff to the Canada Revenue Agency, the Min- 
istry of Finance's Collections Branch should: 

• ensure that temporary staff hired to com- 
pensate for the loss continue to have the 
appropriate skill set and experience to carry 
out collection duties effectively; 

• reassess whether senior collectors, in addi- 
tion to their regular responsibilities, will 
be able to carry out required field visits 
effectively and on a timely basis, and attend 
hearings for the possible suspension of liquor 
and motor vehicle dealer licences, especially 
given their recent significant increase in 
caseloads; and 
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• evaluate the use of private-sector col- 
lection agencies for certain aspects of its 
collections function. 

In the longer term, the Branch should assess 
whether its current permanent staff comple- 
ment is sufficient to maximize the collection of 
non-tax receivables should its mandate expand 
to include all government non-tax receivables. 



| MINISTRY RESPONSE 




The Branch is proactively addressing staffing for 
its accounts receivable function in addition to 
the Retail Sales Tax wind-down. Seventy-four 
temporary staff are being recruited to address 
the remaining accounts receivable inventory 
to maximize collection efforts and recoveries. 
Work will be rebalanced by December 2012. 
The temporary staff will be in place until 
March 2014. 

The Branch had previously considered the use 
of private-sector collection agencies and deter- 
mined that it was not a cost-effective or viable 
option due to privacy issues and the requirement 
for legislative amendments. As part of ongoing 
discussions with its inter-jurisdictional col- 
leagues, it will explore the option of strategic use 
of private-sector collection agencies. 

As the Branch continues to pursue the collec- 
tion of non-tax receivables, staff resourcing will 
be a key consideration and a component of the 
implementation plan. 



Account Prioritization 

The Ministry's Collections Branch primarily uses 
two stand-alone information systems. Corporations 
Tax (CT) and Self-employed Health Tax (SEHT) 
accounts reside in the Branch's older legacy sys- 
tem, the Integrated Collections System (ICS). The 
Branch's collection responsibility for the remain- 
ing tax statutes — for example, Retail Sales Tax, 
Employer Health Tax and Tobacco Tax — is admin- 



istered through a relatively new system called Ont- 
Tax. Prior to 2008, ICS was the only system used 
for the administration of tax collection. In 2008, 
past collection activity for all statutes except CT 
and SEHT was migrated onto OntTax. The Branch 
expects to decommission ICS once it has resolved 
CT amounts owing. OntTax will continue to support 
the collection and administration of the remaining 
statutes, and it will be considered in supporting 
the collection of the province's non-tax amounts 
owing if the Branch's role is expanded as proposed 
in the budget. 

An effective method of prioritizing accounts 
based on the collection risk is important in deter- 
mining the appropriate collections procedures. 
For example, high-risk accounts should be subject 
to more aggressive action, whereas a more mod- 
erate approach could be used for accounts that 
are likely to be paid. With the implementation 
of OntTax and its risk-scoring methodology, the 
Branch has significantly improved how it prioritizes 
accounts held in this system and assigns them to 
the appropriate level of collectors. The Branch has 
also recently implemented a tool called the Flex- 
ible and Integrated Risk System (FAIRS), which 
analyzes an account's collection history to predict 
the likelihood of payment. This helps determine 
the extent of collection efforts that should be 
committed to that account. For accounts that are 
closed, FAIRS provides an overview of which col- 
lection activities were successful in recovering tax 
amounts owing. Currently, FAIRS is used only on 
RST accounts; the Branch plans to start using the 
tool for the remaining accounts housed in OntTax 
in 2013. Since ICS does not have formal risk-scoring 
capability, accounts housed in ICS for the most 
part continue to be prioritized and assigned to col- 
lectors on the basis of dollar value and date of entry 
into collections. 

Through its risk-scoring function, accounts in 
OntTax are prioritized using criteria such as the 
amount owing, number of times or the length of 
time in collections, whether there is a history of 
broken promises, and if any legal actions have 
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been taken on the account. Once the priority of 
an account has been established, it is assigned 
to collectors in one of three tiers: Tier 1 (low- 
risk accounts) and Tiers 2 and 3 (medium- and 
high-risk accounts) . However, we noted that once 
accounts are assigned, collectors often do not use 
the risk ranking to determine the order in which 
they work on the accounts. 

Since the development of the risk-scoring 
methodology in OntTax in 2008, the Branch has 
not updated or formally evaluated it to determine 
whether this tool prioritizes accounts appropriately. 
We noted several anomalies in the scoring that 
indicate a need to review and update the criteria 
so that collectors can make better use of it to set 
priorities. For example, risk scoring assesses some 
of the criteria noted above over the entire history 
of the account while other criteria are assessed 
only after 2008, when risk scoring was adopted in 
OntTax. Also, inconsistency in collection practices 
affects the score assigned to accounts. For instance, 
the risk-scoring methodology assigns points to 
an account every time a notice is sent out. From 
our review of accounts, we noted a fair amount of 
inconsistency in the number of notices sent by col- 
lectors. As a result, more points had been assigned 
to some accounts simply because the collector sent 
more notices; the points assigned did not neces- 
sarily reflect the risk associated with the account. 
We also noted several significant fluctuations in 
the risk scores without a significant change in the 
circumstances of the account that would explain 
the difference. 



RECOMMENDATION 3 



To ensure the effectiveness of its risk-ranking 
methodology for prioritizing collection efforts, 
the Ministry of Finance's Collections Branch 
should formally assess this methodology to 
determine whether it is ranking accounts for 
action appropriately and consistently. The 
Branch should then develop guidelines to 
encourage collectors to use the risk scoring to 
prioritize their work. 



The Branch's Business Enhancement Unit 
and the staff in the unit who work with the 
Ministry's automated risk assessment system 
(FAIRS) will work together to address the Aud- 
itor General's concerns. The Branch will review 
its current methodology and guidelines and 
make any changes to ensure that collectors pri- 
oritize their work in consideration of risks and 
in ways that will maximize the recovery of taxes 
owed to the province. The initial review is to be 
completed by March 31, 2013. 



OVERSIGHT OF COLLECTION ACTIVITIES 

Sufficient oversight of collectors' work is paramount 
in ensuring that their efforts are timely and pro- 
gressive, and that all tools available have been used 
to recover the most money possible. Supervisory 
oversight of collection activities consists of two 
main procedures: 

• Semi-annually, collection managers review 
a sample of each collector's active files. 
Managers also review specific accounts that 
collectors have deemed uncollectible to 
ensure that all collection efforts have been 
exhausted, or where approval is required for 
measures such as initiating payment arrange- 
ments beyond the collector's authority or 
initiating and enforcing legal actions. 

• The Branch's Performance Analysis Unit 
(PAU) and the Ministry's Internal Audit 
Division review accounts that are submitted 
for write-off to ensure that the amounts are 
correct, that policies and procedures were 
followed, and that all collection efforts have 
been exhausted. 

We noted that the managers' semi-annual 
reviews identified specific issues such as the 
timeliness of collection actions, incomplete docu- 
mentation, and failure to use all available tools 
in the collection effort. Although the managers 
informed us that they discussed these issues with 
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the respective collectors, we found no evidence of 
any required action or follow-up to ensure that col- 
lectors were making the required changes. We also 
noted that there is no overall analysis and related 
feedback on any systemic concerns identified 
through the managers' semi-annual reviews or the 
PAU review of accounts submitted for write-off. 



RECOMMENDATION 4 



To ensure that collection efforts are appropriate, 
timely and in compliance with established pro- 
cedures, the Ministry of Finance's Collections 
Branch should ensure that collectors document 
any follow-up action taken in resolving issues 
identified during reviews of their work. The 
Branch should also identify any systemic con- 
cerns, as well as best practices, from its ongoing 
reviews of active files as well as accounts that 
are submitted for write-off. 



The Branch has since introduced a requirement 
for management to perform a follow-up review 
with collectors to ensure that collectors make 
the required changes recommended by manage- 
ment. In addition, the Branch will implement 
a more formalized process to regularly identify 
any systemic concerns and best practices, and 
will work closely with the Business Enhance- 
ment Unit and Performance Analysis Unit to 
improve, communicate and monitor the imple- 
mentation of best practices. 



Management Reports 

In order to oversee collection activities effectively, 
managers should have access to sufficient and 
timely operational and performance informa- 
tion. However, we found that reports produced 
by the Ministry's information systems did not 
adequately support the oversight of the collection 
function. Specifically: 



• ICS, which at the time of our audit was used 
to administer more than 50% of the Branch's 
accounts-receivable inventory, can readily 
produce only pre-programmed "canned" 
reports. Supplemental reports for basic 
information, such as requests for listings of 
accounts with payment plans and legal actions 
in place, take an inordinate amount of time 

to produce. Such reports can be produced 
only through special requests or "tickets" to 
extract the required data from the system — a 
costly and time-consuming process. Although 
the Ministry plans to decommission ICS once 
most of the CT accounts are resolved, the sys- 
tem's inability to produce information useful 
for collections activities will pose problems in 
the meantime. 

• Although the reporting of key information 
on accounts within the OntTax system is 
substantially better than within ICS, there 
are still some limitations. Specifically, the 
amount of the assets on which liens and war- 
rants have been registered prior to the 2008 
implementation of OntTax and that have not 
been renewed or amended since are reported 
at $0 even though information exists within 
the system on the actual amount of the assets 
on which individual liens and warrants have 
been registered. Therefore, the Ministry 
cannot determine the portion of the tax- 
receivable inventory that is secured by liens or 
warrants for these accounts. As well, payment 
plans often do not cover the total amount of 
the debt owing, even though OntTax reports 
the total amount of the debt as being covered 
by the plan. Accordingly, for these situations 
OntTax does not provide an accurate picture 
of the tax-receivable inventory that is secured 
by payment plans. OntTax also does not 
provide details such as the number, amount 
and frequency of instalments associated with 
individual payment plans. 
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RECOMMENDATION 5 



In light of the fact that the OntTax system 
will continue to support the collection and 
administration of the remaining tax statutes, as 
well as the collection of the province's non-tax 
amounts owing if the Branch's role is expanded, 
the Ministry of Finance's Collections Branch 
should work with ministry systems staff to 
ensure that the system reports provide com- 
plete, accurate and up-to-date information on 
debtors' accounts. 



I 
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The Branch's Performance Analysis Unit and 
Business Enhancement Unit will continue to 
work closely with ministry systems staff to 
ensure that the system reports are complete 
and accurate and will address any concerns in a 
timely manner. 



Performance Measures 

The overall mandate of the Ministry's Collections 
Branch is to ensure efficient and effective collec- 
tion of outstanding tax liabilities. The Branch does 
provide some information on its results for a given 
year — for example, amounts collected and number 
of accounts cleared or written off, among other 
things — and compares the results to those of the 
prior year. However, prior to the 2010/11 fiscal 
year, the Branch had not set meaningful bench- 
marks aligned with its mandate against which it 
could evaluate performance, either for the Branch 
or for individual collectors. In the 2010/11 fiscal 
year, the Branch established an overall bench- 
mark performance measure: the total number of 
accounts resolved as a percentage of new accounts 
received. Based on the prior year's result and other 
assumptions, the Branch set a 75% benchmark for 
this measure up to the 2013/14 fiscal year. While 
we recognize the Branch's attempt to establish 
an overall performance measure, we questioned 



whether the resulting measure was meaningful 
given the following: 

• It tracks and reports only on results for RST 
and other statutes that are administered in the 
OntTax system — less than half of the Branch's 
accounts-receivable inventory. Therefore, 

the measure does not provide a complete and 
accurate representation of the effectiveness of 
the Branch's overall collection effort. 

• The number of RST referrals continues 
to decrease substantially given the recent 
transfer of the administration of RST to the 
CRA. This decrease prevents the Branch 
from making any meaningful comparison 
of actual performance with the benchmark. 
For instance, for the fiscal year 2011/12, the 
Branch reported that the number of accounts 
resolved as a percentage of new referrals was 
158%. The Branch significantly exceeded its 
target of 75%, but this could have been due 
to a significant decrease in new delinquent 
accounts rather than a significant increase in 
the collection of existing accounts. 

Individual collector performance is measured 
primarily by the number of clearances (defined as 
the full recovery of debt) . The clearance is credited 
to the last collector assigned to the account with- 
out considering which actions by which collector 
actually led to the recovery of the debt. This also 
gives no credit to collectors who were successful 
in collecting at least part of the outstanding debt. 
The timeliness of collection activities is measured 
by whether accounts with balances greater than 
$100,000, which make up about 80% of the 
Branch's total receivables, are subject to some sort 
of action every 90 days. However, the appropriate- 
ness or effectiveness of the collections action taken 
is not routinely assessed. 

Our research on effective collection practices 
in the private sector indicated that dollar recov- 
ery rate is the key consideration in grading the 
performance of agencies responsible for collec- 
tions, and that different parameters can be used 
to calculate the rate. For instance, recovery rates 
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can be calculated on the entire portfolio of taxes 
receivable, on closed accounts, on accounts by age, 
or by region and district office. Benchmarks can be 
set for the different recoveries and then compared 
to the actual rates. In addition, reporting on the 
reasons why accounts are closed or written off can 
provide valuable insight on the effectiveness of 
internal processes. 



RECOMMENDATION 6 



To enable it to better track the effectiveness of 
its collection efforts, the Ministry of Finance's 
Collections Branch should have more clearly 
defined benchmarks and performance measures 
for collection, both for the Branch itself and for 
individual collectors. The outcomes should be 
tracked, evaluated against established bench- 
marks, and reported periodically. 




The Branch has recently hired a consulting 
firm to assist in developing a balanced per- 
formance measures scorecard for the Branch 
and for its individual collectors. The Branch 
will build on the consultation work provided 
by identifying data elements available from 
ministry systems and establishing performance 
measures and benchmarks. The Branch will 
also attempt to benchmark itself against similar 
organizations and will be discussing this with 
its inter-jurisdictional partners in September 
2012. The initial review is to be completed by 
March 31, 2013. 



Chapter 3 
Section 




Ministry of Training, Colleges and Universities 

University Undergraduate 
Teaching Quality 



Background 




Ontario's 20 publicly assisted universities offer 
graduate and undergraduate degree programs in a 
wide variety of fields. In 2010/11, these universities 
enrolled the equivalent of about 390,000 full-time 
students, excluding about 44,000 foreign and other 
students taking courses not eligible for provincial 
assistance. These universities employed approxi- 
mately 15,000 full-time faculty members. Faculty 
include tenure-stream staff, who have both teach- 
ing and research responsibilities; teaching staff, 
who generally have no research responsibilities; 
and part-time sessional instructors, who are under 
contract to teach one or more courses. 

Most Ontario universities were established 
or continued by acts of the provincial legislature 
that set up their governing structures. University 
governance is often a shared responsibility between 
the Board and the Senate. The Board is generally 
responsible for the university's corporate side, 
including management of property, revenues, 
expenditures and other business affairs. The Sen- 
ate is responsible for academic matters such as 
determining the courses of study, setting admission 
standards, and awarding diplomas and degrees. 

Although historically Ontario universities 
have enjoyed a great deal of autonomy, they are 



generally required to report on financial and other 
matters to the Ministry of Training, Colleges and 
Universities (Ministry) . Also, in addition to tuition 
fees, funds from student services, donations and 
other miscellaneous revenue, universities depend 
on a significant amount of government funding. 
Excluding government funding for student assist- 
ance, research and capital grants, the Ministry 
provided approximately $3.3 billion in operating 
funding to Ontario's universities in 2010/11. 

The Ministry's vision for post-secondary 
education is to build the province's competitive 
advantage and enhance its quality of life by having 
the world's most educated people and the most 
highly skilled workforce. Since the Ministry expects 
that 70% of all new jobs will require education and 
training beyond the high school level, its goal is 
to have 70% of the people of Ontario attain post- 
secondary credentials by 2020. 

University professors typically have two main 
responsibilities — teaching students and conducting 
research in their field of expertise. At the under- 
graduate level, instruction is more oriented to the 
classroom teaching of specific courses, while at the 
graduate level it tends to be done on more of a one- 
to-one basis or in small group settings. Teaching 
students contributes significantly to their learning, 
and ineffective teaching can have an adverse impact 
on learning. Consequently, high-quality instruction 
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can help the Ministry, the universities and Ontario's 
students achieve their goals. 



Audit Objective and Scope 



The objective of this audit was to assess the extent 
to which the Ministry and selected Ontario uni- 
versities support, assess and periodically report 
meaningful performance information on the quality 
of instruction provided to undergraduate students. 

Senior management at the Ministry and selected 
universities reviewed and agreed to our audit 
objective and associated criteria. 

Our audit work was primarily conducted at 
Brock University, the University of Ontario Insti- 
tute of Technology, the University of Toronto, and 
the Ministry of Training, Colleges and Universities. 
This work focused mainly on students at the under- 
graduate level, because undergraduates make up 
the majority of university students. We also met 
with the Council of Ontario Universities (COU), 
which represents the interests of Ontario's publicly 
assisted universities; the Ontario Confederation of 
University Faculty Associations, which represents 
the faculty associations of Ontario universities; 
and the Ontario Undergraduate Student Alliance, 
which represents the interests of undergraduate 
university students. We also spoke to representa- 
tives of the Higher Education Quality Council of 
Ontario, an Ontario government agency set up to 
generate evidence-based research to assist in the 
improvement of the post-secondary education 
system, and to the Ontario Universities Council 
on Quality Assurance (OUCQA). The OUCQA was 
established by the COU to oversee quality assur- 
ance processes for all academic programs offered 
by Ontario's publicly assisted universities. 

In conducting our audit work we reviewed 
relevant legislation, policies and procedures, and 
met with appropriate staff of the Ministry and the 
universities we visited, including faculty members, 
department heads and deans (generally, a dean is 



in charge of a professional faculty such as engineer- 
ing or law, or an academic faculty such as human- 
ities or social sciences) . In addition, we met with a 
number of students at each university we visited, 
researched practices in other jurisdictions, and 
engaged an adviser who is recognized as a leading 
scholar in research on student development and 
teaching assessment. 

We did not rely on the Ministry's or universi- 
ties' internal audit departments to reduce the 
extent of our audit work, because they had not 
conducted any recent audit work on university 
teaching quality. 



Summary 




From the Ministry's perspective, a university's most 
important mandate is that it does a good job of 
teaching its students and preparing them for the 
future workforce. We believe students, their parents 
and the public would agree. 

The administrators (deans and faculty or depart- 
ment heads) we spoke to at the universities visited 
noted that assessing undergraduate teaching qual- 
ity in universities is complex and not easily quantifi- 
able. Nevertheless, most felt that relevant measures 
could be developed to give insight into teaching 
quality. Although both the Ministry and the univer- 
sities we visited were not formally assessing and 
reporting on teaching quality and effectiveness on 
a regular basis, we observed that information con- 
cerning teaching performance that may be useful 
in some capacity was available at all three universi- 
ties. For instance, all Ontario universities, including 
the ones we visited, have instituted a good process 
whereby undergraduate students are generally 
given the opportunity to complete evaluations 
toward the end of the courses they take. 

However, our review indicated that little aggre- 
gate analysis of the student evaluations is done at 
the universities we visited, and only about one- 
quarter of Ontario's universities indicated that they 
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make the summarized results of these evaluations 
available to students to assist them in their course 
selection decisions. 

It should be acknowledged that, notwithstand- 
ing any issues regarding teaching quality, a recent 
survey of Ontario university students indicates that 
78% of senior-year students were satisfied with 
their educational experience, and all three of the 
universities we visited had processes in place to 
varying degrees to improve teaching quality, such 
as establishing teaching centres and considering 
teaching performance in tenure and promotion 
decisions. Nevertheless, some of our key observa- 
tions on teaching quality at the universities we 
visited and from available data at other universities 
suggest a need to better ensure that teaching qual- 
ity is valued, encouraged and rewarded. 

These observations include the following: 

• Only one of the universities we visited peri- 
odically aggregated undergraduate student 
course evaluation data, although one of the 
other universities informed us that it was 
in the process of establishing a new course 
evaluation framework and planned to aggre- 
gate course evaluation results and analyze 
them once the system was fully implemented. 
The third university we visited was not aggre- 
gating and reporting the results of student 
evaluations, nor did it plan to. In fact, we 
were advised that a decision to defer a faculty 
member's tenure was reversed on appeal in 
part because this university's administrators 
did not have aggregate student evaluation 
data to demonstrate that the results of this 
professor's student evaluations were below 
their expectations. 

• A number of faculty we spoke to felt that 
their annual performance appraisals did not 
provide them with appropriate feedback on 
teaching performance. We noted examples 
where student evaluations had been critical 
of teaching performance but there was no 
evidence that specific guidance was provided 
or that the faculty members had sought 



assistance to improve their teaching skills. 
Also, none of the universities visited required 
providing written performance appraisals 
to sessional instructors, even though they 
accounted for between 10% and 24% of 
full-time-equivalent staff. 

• We reviewed a sample of tenure and promo- 
tion decisions at the three universities visited 
and noted that the documentation available 
on teaching quality was generally positive. 
However, at one university we were able to 
review documentation relating to 2011 only, 
because documentation for previous decisions 
had been destroyed in accordance with the 
university's collective agreement. At another 
university, one individual who was granted 
tenure was rated well below average in stu- 
dent course evaluations, and administration 
had consistently suggested improvement in 
teaching performance. This university did not 
have the option to defer tenure decisions until 
performance improved, because a negative 
tenure decision leads to termination accord- 
ing to its agreement with faculty. 

• According to aggregate undergraduate stu- 
dent course evaluation results at one of the 
universities visited, the university's Faculty of 
Education consistently outperformed all other 
faculties by a significant margin in overall 
ratings of teacher effectiveness. We were 
told that virtually all members of this faculty, 
unlike those of other faculties, had formal 
training in how to teach. Ontario universities 
in general do not require faculty members to 
have formal training in teaching. At all three 
universities visited, performance appraisals 
revealed examples where, to varying degrees, 
there was room for improvement in teaching 
effectiveness, but we did not see evidence 
that these instructors had been provided with 
specific guidance or sought assistance from 
the universities' teaching and learning cen- 
tres. In fact, at two of the universities visited, 
records at the centres indicated that faculty 
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attendance at teaching workshops averaged 
less than one hour per instructor per year. 

• While ministry funding to Ontario universi- 
ties has increased significantly over the past 
decade, according to the Council of Ontario 
Universities (COU), Ontario universities 
receive less government funding per student 
than universities in any other Canadian 
province. A recent COU study also noted that 
Ontario's student-to-faculty ratio is the high- 
est in Canada. In one university visited, 75% 
of first-year classroom spaces in several facul- 
ties were in classes with over 100 students. 
Research indicates that large classes can pose 
challenges to providing effective instruction, 
which can impact student learning. In a recent 
survey, faculty across the province noted that 
their workload had increased over the last five 
years, resulting in their reducing one-on-one 
time with students outside the classroom, 
giving more multiple-choice tests and giving 
fewer writing assignments. 

• The Ministry is making progress toward 
achieving its goal of having 70% of Ontario's 
population hold post-secondary credentials by 
2020. It reported in 2011 that 64% of Ontar- 
ians aged 25 to 64 had post-secondary cre- 
dentials, compared to 56% in 2002. However, 
we found that two years after graduation only 
65% of graduates surveyed were employed 
full-time in a job that was related to the 
skills acquired in their studies. The Ontario 
Undergraduate Student Alliance indicated to 
us that students would find such information 
on graduate employment outcomes beneficial 
in choosing their university and program 

of study. 



PROCEDURES TO ASSESS 
TEACHING QUALITY 

Although the administrators we spoke to at the 
universities we visited (such as deans and faculty 
department heads) noted that assessing under- 
graduate teaching quality is complex and not easily 
quantifiable, most felt that relevant measures could 
be developed to give insight into teaching quality. 
Administrators also noted that no one indicator 
was sufficient in itself to assess teaching quality. 
Although the universities we visited were not for- 
mally assessing and reporting on teaching quality 
and effectiveness on a regular basis, we observed 
that information on teaching performance that may 
be useful in some capacity — such as student course 
evaluations, performance appraisals of instructors, 
student surveys and program reviews — was avail- 
able at all three universities. 

Student Course Evaluations 

Student course evaluations are questionnaires that 
require a written or selected response on a wide 
range of topics such as course content and the qual- 
ity of instruction. Although the process differed at 
the three universities we visited, questionnaires 
are typically given to students two to three weeks 
before the final course exam, and results are not 
provided to instructors until after the final exam is 
marked and student grades are submitted. In gen- 
eral, policies surrounding the student evaluation 
process were designed to ensure that the instructor 
does not influence the responses and that the 
responses are provided anonymously. 

Research on student evaluations reveals that 
some faculty members question the ability of 
students to properly assess instructors, but other 
research indicates that most have a positive view of 
student course evaluations. Even though research 
shows that students cannot accurately assess an 
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instructor's knowledge or competency in the disci- 
pline, a number of studies conclude that student 
evaluations are valid, reliable and useful when 
examined thoroughly. In addition, these studies 
indicate that students can very accurately assess 
the effectiveness of in-class teaching, including 
presentation, clarity, the instructor's level of organ- 
ization, the course workload and their own overall 
learning experience. 

We reviewed the student evaluation process 
at the three universities and were informed that 
most undergraduate courses were being evaluated. 
Exceptions were small classes and one university 
that did not always evaluate courses taught by 
part-time sessional staff. Only one of the three 
universities used a common questionnaire for all 
undergraduate courses. The other two universities 
did not have a common student evaluation ques- 
tionnaire to be used across different faculties and, 
in one case, student evaluation questionnaires even 
differed across departments within a single faculty. 

The university that used a common question- 
naire had also taken the initiative of developing 
an online system where data could be aggregated 
and compared at the faculty and university levels. 
Our analysis of this data revealed that, over the last 
five years, the average instructor score was 1.22 on 
a scale of -2 to +2, indicating that students were 
generally satisfied with the quality of instruction 
they had received. Further analysis of this data 
revealed the following: 

• There were significant differences in pro- 
fessors' scores between faculties, with the 
average scores over the last five years ranging 
from 1.03 to 1.46. However, the university 
had not formally investigated reasons for dif- 
ferences between faculties in order to identify 
and share best practices that could improve 
teaching quality at lower-performing faculties. 

• We also noted differences in the performance 
of different types of instructors. For example, 
over the last three years, tenure-stream and 
teaching-stream faculty were rated almost 
identically (1.23 and 1.25) by students, 



whereas sessional instructors on average 
received lower scores (1.03). This university 
relies heavily on the use of sessional staff, who 
accounted for 24% of its full-time-equivalent 
staff and were responsible for teaching 
approximately 40% of its courses. This overall 
student assessment was consistent with the 
opinions of the administrators we spoke to, 
who indicated that, partly because sessional 
staff often have competing employment com- 
mitments, they generally provided instruction 
that was of lower quality. 
One of the other universities visited informed 
us that it was in the process of establishing a new 
online course evaluation framework with core 
questions common to all faculties and departments. 
As well, this university planned to aggregate the 
results and undertake analysis of the data once the 
system was fully implemented. At this university, 
the vast majority of faculty received a teaching rat- 
ing of at least "good" in the student course evalua- 
tions we reviewed. 

We could not review student course evaluations 
at the third university because its collective agree- 
ment with faculty stipulates evaluations to be the 
property of the professor, and we were not given 
access to them. The collective agreement stipulates 
that student evaluations can be used by administra- 
tion in the tenure and promotion process, and for 
annual appraisals. However, a score from student 
evaluations was recorded in about half the activity 
reports we reviewed (faculty members submit these 
reports to administrators for annual appraisal pur- 
poses) . Most of these scores were positive, although 
we could not verify their accuracy, and in many 
cases it was unclear whether the score represented 
an overall average of different questions concern- 
ing teaching performance, or whether it reflected 
the answers to a single question. We observed an 
instance at this university where the decision to 
defer a faculty member's tenure, based on concerns 
about teaching, had been reversed on appeal. Uni- 
versity administrators informed us that this was due 
in part to the fact that they did not have aggregate 
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student evaluation data to demonstrate that the 
results of this professor's student evaluations were 
below their expectations. 

We found that only one of the three universities 
gave students access to the summarized results of 
course evaluations, and this university informed us 
that only half of its students could easily access the 
results. This university also informed us that once 
its new framework is implemented, students would 
have access to the summarized course evaluation 
results unless the instructor elects to opt out of 
the process. The students we spoke to at all three 
universities, as well as representatives from the 
Ontario Undergraduate Student Alliance (OUSA), 
felt that students would find the results of student 
course evaluations useful for making decisions 
regarding future course selection. As well, OUSA 
representatives and some students we spoke to 
noted that when results are not available, students 
tend to not take the evaluation process as seriously. 
This was because they feel that the results are not 
actually used by the university or the professor. 
We contacted all 20 publicly funded universities in 
Ontario and were informed that they all provide 
most students the opportunity to complete course 
evaluations. However, just five indicated that the 
results of these evaluations are generally made 
available to students. 

While administrators at two of the three uni- 
versities generally felt that the results of student 
course evaluations could be used along with other 
indicators to provide insight into teaching quality, 
none of the three universities formally used such 
evaluations in aggregate to periodically analyze 
and report on teaching quality at the level of the 
university as a whole or at the faculties we focused 
our work on. 

Annual Performance Appraisals of Faculty 

The annual appraisal of faculty can assist faculty 
members in their career development and identify 
weaknesses in performance to facilitate improve- 
ment. The three universities we visited had policies 



in place to annually assess the teaching perform- 
ance of tenure-stream and teaching-stream faculty. 
Although the three institutions used different 
processes, all three included a requirement for 
faculty members to submit an activity report to 
the administration detailing, among other things, 
their teaching activities. This activity report could 
include information on courses taught, supervision 
of students, course development, awards and 
honours, improvements in teaching, and profes- 
sional development. We were informed that these 
activity reports and other relevant information are 
considered along with student course evaluations 
in the appraisal of the faculty members' overall and 
teaching performance. The primary source of data 
available to assess in-class teaching performance 
at the three universities we visited was student 
course evaluations, because in almost all cases 
these were the only documented assessments of 
classroom instruction. 

At one of the universities visited, administrators 
gave the vast majority of faculty selected for review 
an overall rating indicating satisfactory teaching 
performance. At the second university, although 
the faculty members we selected had received 
an annual appraisal, about half the professors 
evaluated did not receive a quantitative score or 
a qualitative rating tied to specific performance 
criteria. At the third university, the vast majority of 
faculty were appraised as being in compliance with 
their collective agreement — which, according to 
the agreement, implies satisfactory teaching per- 
formance. The administration at this university was 
generally of the opinion that recognizing differing 
levels of performance would be useful in facilitating 
improvements in teaching. However, under the col- 
lective agreement, there was no requirement to pro- 
vide faculty members with a performance appraisal 
unless a performance-related concern arose. 

We reviewed a sample of faculty performance 
appraisals at the three universities visited and 
noted the following: 

• In a few instances at one university, student 
evaluations suggested that there may be room 
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for improvement in teaching performance, 
but there was no evidence in the appraisal 
or in the file that specific guidance had been 
provided to the faculty members or that any 
action had been taken by the faculty members 
to facilitate improvement. 
Another university's institution- wide student 
evaluation results indicated that its students 
were not fully satisfied with the performance 
of approximately 20% of full-time faculty 
members. Yet, despite the suggestion in 
student evaluations that there was room for 
improvement, in some instances we saw no 
evidence that the faculty member either had 
been directed to seek or had sought help. We 
also observed instances where the administra- 
tion's appraisal of satisfactory performance 
was not in line with the results of student 
course evaluations. In one such case, a faculty 
member's student evaluation results were well 
below the average, and the administrator had 
recommended that this individual seek help 
with teaching skills, but, nevertheless, this 
faculty member's teaching performance was 
appraised as eight out of 10. 
At the third university, student comments 
recorded on course evaluations were not 
generally reviewed, and the administration 
often evaluated only the one question on the 
student evaluations that asks about overall 
performance. This was despite the avail- 
ability of student responses to a number of 
potentially informative questions, including 
whether the instructor presented material 
in an organized and well-planned manner, 
explained concepts clearly with appropriate 
use of examples, and communicated enthusi- 
asm and interest in the course materials. In 
our review of a sample of student evaluations, 
in some cases it was not evident whether the 
administration had provided feedback to fac- 
ulty members whose teaching in specific areas 
likely had room for improvement (although 
the vast majority of faculty members received 



an overall score qualitatively associated with 

good performance). 
At the three universities we visited, a number of 
professors told us that student course evaluations 
factored too highly in the assessment of teaching 
and that their performance appraisals did not 
provide them with appropriate feedback on their 
teaching. Some professors at one university sug- 
gested that other evaluation techniques, such as 
periodic peer reviews by other professors, could be 
used to augment student course evaluations of in- 
class instruction. However, none of the universities 
visited used these other evaluation techniques to do 
so on a regular basis. 

Performance Appraisals of 
Sessional Instructors 

As a general rule, none of the universities visited 
provided written annual performance appraisals 
to their sessional instructors. Sessional instructors 
at the three universities accounted for between 
10% and 24% of full-time-equivalent faculty. At 
the university that had the largest proportion of 
sessional staff, many of them had been employed at 
the university for several years. 

The sessional instructors we sampled generally 
received favourable evaluations from students at 
two of the three universities visited. At the third 
university, we observed several instances where ses- 
sional instructors received lower course evaluation 
scores than the faculty average, and there was no 
evidence that university administrators instructed 
these faculty members to seek assistance. In one 
case a sessional instructor received a contract to 
teach full time after consistently receiving some of 
the poorest student evaluations in three successive 
years. While we were informed that the administra- 
tion had referred this individual to the university's 
teaching and learning centre, our review of the 
centre's attendance log did not indicate attendance 
at any workshops by this instructor during the pre- 
vious two years, and there was no documentation 
for any other professional development work done 
at the centre. 



2012 Annual Report of the Office of the Auditor General of Ontario 



Because two of the three universities did not 
provide annual performance appraisals to their ses- 
sional instructors, and one did so only occasionally, 
documentation concerning the teaching perform- 
ance of sessional staff was generally limited to the 
results of student course evaluations. This was of 
concern, as research suggests that no one source of 
information is sufficient to assess teaching quality 
and that student course evaluations are just one of 
the factors to consider. As well, one university had 
no requirement to administer student evaluations 
if the course was taught by a sessional instructor. 
Consequently, for some sessional staff, there 
may be nothing on file to indicate the quality of 
their teaching. 

Student Surveys 

All three of the universities we visited participated 
in national student surveys that gather information 
to gain insight into the students' undergraduate and 
graduate school experiences. The National Survey 
of Student Engagement (NSSE) was the most-used 
survey and applicable to the widest cross-section of 
students. The NSSE is described as the leading tool 
used to measure student engagement in universi- 
ties, and there is significant literature associating 
student engagement with learning outcomes. While 
the NSSE survey asks a lot of informative questions, 
those responsible for reviewing and analyzing the 
results at the universities we visited generally felt 
that the results of the NSSE could be used only 
to a limited degree to assess teaching quality. 
Consequently, they were not using the results for 
such purposes. 

Nevertheless, we found that the NSSE solicits 
information that can inform universities of areas 
where improvements can be made. To illustrate, at 
two of the universities we visited, between 25% and 
31% of first-year students selected reducing class 
size as an item the university needs to address to 
improve their learning experience. Our discussions 
with one of the survey's creators identified that 
NSSE results are more useful at the faculty and 



department levels than at the university level to 
identify specific areas that require further scrutiny 
or action. 

Overall, NSSE survey results indicate that 
students were generally satisfied with their educa- 
tional experience. For example, in 2011, between 
71% and 86% of senior-year students at the three 
universities rated their educational experience as 
either good or excellent. The Ontario average was 
78%. In comparison, American peer institutions 
had an average of 84%. 

All three universities had also participated in the 
2010 Canadian Graduate and Professional Student 
Survey, in which students were asked, among other 
things, to comment directly on graduate-level 
teaching quality. Although none of the universities 
visited had formally used this survey to identify and 
address weaknesses, we observed that in aggregate 
all three performed close to the Ontario average in 
questions that directly concern teaching quality. For 
example, for all Ontario universities, an average of 
87% of students surveyed rated the overall quality 
of graduate-level teaching as good to excellent, and 
all three universities we visited received similar 
results. At the undergraduate level, two of the three 
universities we visited participated in the Canadian 
University Survey Consortium's 2009 survey and 
again performed close to the average with respect 
to teaching quality. Overall, 90% of students sur- 
veyed were generally satisfied with the quality of 
teaching they received, and the two universities 
scored 88% and 93%, respectively. 

Program Reviews 

In 2010, the Council of Ontario Universities (COU) 
established the Ontario Universities Council on 
Quality Assurance (OUCQA). The OUCQA was 
established to oversee quality assurance processes 
for all academic programs offered by Ontario's 
publicly assisted universities and to bring the 
quality assurance processes that had been in exist- 
ence for many years under a common framework. 
The OUCQA is also responsible for auditing each 
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university's quality assurance processes and approv- 
ing new graduate and undergraduate programs. 
We were informed that all universities adhere to 
this quality assurance framework and that ministry 
funding for new programs is conditional on the 
programs' approval by the OUCQA. 

The emphasis of the quality assurance frame- 
work is on evaluating the overall quality of pro- 
grams offered by universities. As part of this process, 
universities are required to self-assess their existing 
programs once every eight years against a set of 
criteria that includes an evaluation of the program's 
curriculum, teaching and assessment methods, and 
human and financial resources. As well, universities 
are required to assess whether student learning 
outcomes have been achieved. Learning outcomes 
include not only the acquisition of specific know- 
ledge in a given field, but also general competencies 
such as the ability to think critically, communicate 
effectively and work well with others. 

Program reviews offer the possibility of provid- 
ing at least indirect feedback on teaching quality, 
since teaching contributes significantly to student 
learning outcomes. However, discussions with 
university staff and our examination of program 
reviews completed under the new quality assur- 
ance framework at two of the universities visited 
revealed that measuring student learning outcomes 
continues to pose challenges for universities, a view 
also shared by representatives we spoke to from the 
OUCQA and the COU. Recognizing this, the COU 
developed a Teaching and Learning Task Force in 
2011 to identify and promote practices to improve 
teaching and learning outcomes. We were informed 
that the Task Force was currently in the process 
of surveying university administrators to identify 
how learning outcomes are being assessed and to 
gather opinions on how to improve the assessment 
of learning outcomes. 

The program reviews at one of the universities 
identified that it had taken steps to move beyond 
the presumption that offering appropriate courses 
with appropriate learning objectives would ultim- 
ately lead to students meeting the intended learn- 



ing outcomes. This university had used responses 
from student surveys to illustrate that learning 
outcomes had been achieved. However, student 
perceptions of their learning outcomes may not 
be indicative of actual student achievement. For 
example, although the students surveyed were 
confident of their writing ability, faculty noted their 
concerns with the quality of student essays and 
writing skills. Representatives at this university 
indicated that they were still working on ways to 
measure student learning outcomes, and some sug- 
gested that standardized tests of key competencies 
could assist in this measurement. 

Program reviews, supplemented by independent 
audits of the review process, can provide valuable 
feedback about program quality, and we believe 
that with further refinement they could also pro- 
vide useful feedback on student learning outcomes. 



RECOMMENDATION 1 



To help ensure that administrators and students 
have sufficient information to make informed 
decisions, and that all faculty members receive 
the necessary feedback to maintain or enhance 
teaching quality, universities should: 

• consider means to aggregate student course 
evaluation information at the university, 
faculty and department levels so that admin- 
istrators can identify best practices and areas 
requiring attention; 

• develop a core set of student course evalua- 
tion questions to be used throughout the 
university to facilitate comparison of student 
evaluation results; 

• provide students with the summarized 
results of student course evaluations to 
assist them in making informed decisions on 
course selection; and 

• ensure that faculty, including sessional 
faculty, periodically receive constructive 
feedback on their teaching effectiveness, and 
encourage faculty to undertake any neces- 
sary professional development. 
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The universities generally agreed that both 
university administration and students need 
sufficient information to make informed deci- 
sions and that faculty should receive sufficient 
feedback to enhance their teaching skills as 
appropriate. One university notes that it is 
already well-advanced in the implementation 
of the recommendations, which it expects will 
strengthen both the quality of post-secondary 
education and its accountability. 

Another university stated that it is already 
in the process of implementing an online course 
evaluation framework that utilizes common 
university-wide questions that will be answered 
by all students to ensure consistency and enable 
aggregation and comparability. In addition, 
once implemented, the new system will further 
enhance student access to the results of course 
evaluations and will facilitate constructive, 
comparative feedback for instructors and 
administrators. This university is also commit- 
ted to ensuring that all sessional instructors 
periodically receive constructive feedback on 
their teaching effectiveness. 

The third university noted that it was 
interested in developing a core set of student 
course evaluation questions that could be used 
throughout the university. As well, the univer- 
sity understands the use and value of aggregate 
data and providing students with the results 
of such evaluations. However, to accomplish 
this, the university will need to work with 
its faculty association through the collective 
bargaining process. 



TENURE AND PROMOTION OF FACULTY 

Faculty members referred to as tenured have met a 
university's probationary requirements pertaining 
to teaching, research and other service to the 



university. Tenure-stream faculty are the trad- 
itional means by which universities are staffed and 
represent the largest group of full-time-equivalent 
faculty at all three universities visited. Their 
responsibilities generally consist of 40% teach- 
ing, 40% research and 20% other service (such 
as participating on committees and performing 
other administrative duties) . At the universities 
visited, faculty members who achieved tenure were 
generally awarded the rank of associate professor. 
Once faculty members receive tenure, it is very 
difficult for the university to terminate them for 
performance concerns, although, as in any job, 
their further progress depends on their ongoing 
performance. Summaries of tenure applications 
over the last three years revealed that the three 
universities granted tenure to 95%, 96% and 100% 
of the applicants, respectively. We were advised 
that competition for tenure-stream positions results 
in numerous well-qualified candidates applying for 
tenure positions, and therefore high success rates 
are to be expected. 

During their probationary period, tenure-stream 
faculty members must demonstrate sufficient com- 
petence in both teaching and research. At the three 
universities visited, the probationary period was 
five to six years. Two of the universities also had 
interim review processes during the probationary 
period to assess and provide feedback on the faculty 
member's progress. At the conclusion of the proba- 
tionary period, faculty members at all three univer- 
sities are to prepare documentation with respect to 
teaching and research, among other achievements, 
in support of their application for tenure. 

While the method of reviewing applications 
for tenure and promotion differed at the three 
universities visited, all three included the review 
of tenure and promotion applications by multiple 
committees, and approval by the university's aca- 
demic vice-president and/or president. Information 
at the disposal of these committees concerning 
teaching included a teaching dossier that typically 
contained a description of the individual's teach- 
ing philosophy, course development work, sample 
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assignments, letters from students, letters from 
colleagues, and relevant honours and awards. The 
summarized results of student evaluations were 
also typically available to the committees. 

Our review of the tenure and promotion 
processes at all three universities revealed that, 
while it was evident what information on teaching 
performance was being provided for the commit- 
tees' consideration, it was unclear to what degree 
this information factored into the committees' 
decisions. We also noted that even though all three 
universities appraise full-time faculty members' 
performance annually, the appraisals were not used 
to inform tenure and promotion decisions at one 
university, and at the other two universities it was 
unclear whether they were used. 

We reviewed a sample of tenure and promotion 
cases at the three universities in detail. At one uni- 
versity, while it was unclear to what degree differ- 
ent sources of information factored into tenure and 
promotion decisions, the documentation submitted 
on teaching quality was generally positive and sup- 
ported committee decisions. We noted from this 
university's record of tenure decisions over the last 
three years that 94% of faculty had achieved tenure 
on the basis of a combination of excellent research 
and competent teaching, and only one faculty mem- 
ber had received tenure on the basis of excellent 
teaching and competent research. While research is 
important to the quality of education provided by 
a university and to its reputation, faculty seeking 
tenure might not be sufficiently motivated to pur- 
sue excellence in their teaching if they believe such 
results to be indicative of the relative value placed 
on teaching in tenure decisions. 

At another university, we were told that student 
course evaluations were heavily relied on for tenure 
and promotion decisions. However, these were not 
available for our review because of restrictions in 
the university's collective agreement. While we did 
not have access to actual student evaluation results, 
a score was generally recorded in the documenta- 
tion made available to us. In the decisions that were 
available for our review, the majority of evaluations 



were positive. However, we could not check the 
actual student evaluations to verify the accuracy of 
such information. 

At this same university, although peer evalua- 
tions were solicited for tenure and promotion deci- 
sions, generally such evaluations did not include an 
in-class evaluation of the prospective candidate's 
instruction as was done at one of the other universi- 
ties visited. The information that we could review at 
this university did not identify significant concerns 
about teaching quality in tenure and promotion 
decisions — although, as previously noted, one 
decision to defer tenure was overturned on appeal 
because the administration did not have sufficient 
information to demonstrate that the faculty mem- 
ber's teaching performance was significantly below 
average. At this university we were able to review 
documentation relating only to the 2011 tenure and 
promotion decisions. This was due to the collective 
agreement, which required all materials relating to 
the tenure and promotion process to be destroyed 
immediately after a decision is reached unless an 
appeal or grievance is lodged. Therefore, the univer- 
sity could not demonstrate that a sound process had 
been followed in reaching any previous decisions. 

The tenure and promotion decisions that we 
reviewed at the remaining university included 
examples where faculty members who received 
student evaluation scores below the average for their 
faculty were granted tenure and/or promotion to 
full professor. In each of these cases, the administra- 
tion noted that the faculty member's research was 
highly rated. These cases included one where, in the 
two years leading up to tenure, students rated the 
faculty member's performance well below average. 
The administration had on several occasions sug- 
gested the faculty member seek out opportunities to 
improve teaching performance. We observed that, 
unlike one of the other universities we visited, this 
university's agreement with its faculty did not give 
it the option to defer decisions on tenure once the 
application was under review. A negative decision 
leads to the faculty member's termination. This 
was of concern, as we noted a number of examples 
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where, after the granting of tenure or promotion, 
student evaluation results declined or did not 
improve substantially. In some circumstances, 
allowing the deferral of tenure decisions could 
give candidates for tenure an incentive to improve 
their teaching and the additional time to take 
formal training. 



RECOMMENDATION 2 



To help ensure that tenure and promotion 
decisions and the underlying documentation 
appropriately reflect the relative import- 
ance of a professor's teaching ability, the 
universities should: 

• ensure that all relevant information on 
teaching performance is made available to 
tenure and promotion committees and that 
all documentation supporting their recom- 
mendations is retained for an appropriate 
period of time; and 

• explore means to ensure that tenure and pro- 
motion processes clearly reflect the relative 
importance teaching ability has with respect 
to such decisions. 



I 



SUMMARY OF UNIVERSITI 
RESPONSES 




The universities supported this recommenda- 
tion and indicated they strongly value their pro- 
fessors' teaching abilities. One university added 
that it will review its policies and processes to 
explore means to provide as much information 
as possible with regard to teaching and other 
duties and responsibilities in order to further 
strengthen existing tenure and promotion 
processes. Another university noted that its cur- 
rent policies address this recommendation, and 
it will endeavour to strengthen these policies 
even further. As well, the university noted that 
it will enhance the communication and training 
for academic administrators and members of 
tenure and promotion committees and put in 
place more specific guidelines for the retention 
of related documentation. 



TRAINING AND PROFESSIONAL 
DEVELOPMENT 

Ontario universities in general do not require fac- 
ulty members to have formal training in teaching 
like elementary and secondary school teachers, 
who are required, at a minimum, to take a one- 
year program of teacher education. In addition, 
the three universities we visited did not require 
faculty members to undertake periodic professional 
development related to teaching. Nevertheless, 
most administrators and faculty members we 
spoke to at all three universities felt that some 
formal training was beneficial to being an effective 
university teacher. 

Research we reviewed on other jurisdic- 
tions identified that initial training of university 
instructors is now common in several countries, 
including the United Kingdom. Some institutions 
in these countries can require 120 to 500 hours 
of teacher training. In some cases, the completion 
of this training factors into tenure and promotion 
decisions. As well, a study done at 22 universities 
in eight countries found that students judged that 
professors who had received training in teaching 
had improved in areas such as enthusiasm, organ- 
izational ability and rapport with students. 

According to aggregate student course evalua- 
tion results at one of the universities we visited, its 
Faculty of Education consistently outperformed all 
other faculties by a significant margin in terms of 
overall professor ratings. Although the university 
had not formally identified the reasons for this 
difference, we were informed that virtually all 
members of this faculty had formal training in how 
to teach. 

At all three universities visited, our review of 
faculty performance appraisals revealed examples 
where there was room for improvement in teaching 
performance, but we did not see evidence that the 
administration provided specific guidance to faculty 
or that faculty subsequently sought out assistance. 
We noted that, while one university had a good 
initiative to identify and review the bottom 10% 
of faculty members based on student evaluation 
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results, the review and any recommended actions 
were informal. Consequently, it was unclear who 
the administration determined needed assistance 
and whether recommendations to obtain assistance 
were made. 

We observed that the three universities visited 
had teaching and learning centres to support, pro- 
mote and advance the quality of teaching at their 
institutions. We also noted that among the services 
provided at all three of these teaching and learning 
centres were one-on-one consultations and a var- 
iety of teaching workshops open to all faculty mem- 
bers. The faculty members we spoke to at all three 
universities generally had a positive opinion of the 
quality of services offered by these teaching and 
learning centres. However, the available attendance 
records at the centres (attendance records are not 
maintained for all services provided) indicate that 
average attendance in teaching workshops per 
faculty member was less than two hours annually at 
one of the three universities and, at the other two 
universities, less than one hour per year. 

All three universities offered orientation to 
newly hired faculty that included topics related to 
teaching. We observed that at one of the universi- 
ties, in addition to a one-day general orientation 
provided to all new employees, orientation aimed 
specifically at new faculty was just four hours in 
length. Attendance records indicate that it was 
attended by less than 25% of new hires in 2011. We 
observed that the other two universities offered far 
more extensive orientation, providing program- 
ming over a full week that covered topics such as 
how to manage the classroom, engage students, 
design assignments, use technology in the class- 
room, and plan and maintain courses. At one of 
these universities, the orientation sessions were 
open to all instructors, including sessional staff, but 
we were unable to accurately estimate attendance. 
At the other university, in which attendance was 
mandatory, we observed that attendance exceeded 
80% of new full-time staff. However, this orienta- 
tion was not open to sessional instructors; a separ- 
ate one-day voluntary orientation was provided to 
them, attended by 60% of new sessional staff. 



We noted that the teaching and learning centres 
at all three universities offered comprehensive 
training to students serving in a teaching capacity, 
including programs that provide a teaching cer- 
tificate upon completion. These programs varied 
in length, with the longest certificate programs at 
each university requiring participation in eight to 
16 workshops over approximately one to two years. 
Similar programs were generally not in place for 
faculty members at the universities, although one 
university informed us that it had recently opened 
its certificate program to all faculty members, 
and another offered a program that awarded a 
certificate to faculty who had completed at least six 
teaching workshops. 



RECOMMENDATION 3 



To help ensure that all faculty members provide 
effective classroom instruction, universities 
should work with faculty to encourage greater 
participation in professional development 
activities and implement procedures to ensure 
that faculty who would benefit from additional 
teacher training are formally encouraged to 
participate in these activities. 



I 




SUMMARY OF UNIVERSITIES 
RESPONSES 



The universities generally concurred with this 
recommendation and noted they will continue to 
emphasize the importance of ongoing develop- 
ment of teaching expertise for all instructors 
through their teaching and learning centres. 
One university further noted that it would put in 
place a mechanism by which those instructors 
who require additional teaching support are 
identified and encouraged to engage in suitable 
professional development. Another university 
agreed with the importance of effective class- 
room instruction and will work with its faculty 
association and faculty to pursue strategies to 
encourage greater participation in teaching- 
specific professional development, particularly 
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for those faculty members who stand to benefit 
from such training. In support of its commitment 
to strong teaching and learning, this university 
created a new position — Vice Provost of Teach- 
ing and Learning — and enhanced the role of its 
Centre for Pedagogical Innovation in order to 
highlight the importance of pedagogy, teach- 
ing and learning. This university also noted 
that it would review its processes to further 
encourage staff to undertake any necessary 
professional development. 



FACULTY COST AN D WORKLOAD 

Total operating revenue for Ontario universities 
amounted to almost $7.5 billion in the 2010/11 
fiscal year, with the Ministry's operating grants 
accounting for a significant portion of this amount. 
Over the last 10 years, ministry assistance to 
universities has almost doubled, increasing from 
$1.7 billion (2000/01) to $3.3 billion (2010/11). 
After adjusting for the effects of inflation, this 
amounts to a 57% increase, which compares to 
a 54% increase in enrolment, from 253,000 to 
390,000 students. While ministry funding to 
Ontario universities has increased significantly 
over the past decade, according to COU analysis 
Ontario universities receive less government fund- 
ing per student than universities in any other Can- 
adian province. Even when tuition is considered, 
ministry and COU analyses show that Ontario uni- 
versities rank at or near the bottom of all Canadian 
universities with respect to revenue received per 
full-time-equivalent student. 

Expenditures on the salaries of faculty holding 
an academic rank, such as tenure- and teaching- 
stream staff, increased from $1.05 billion to 
$2.1 billion between 2000/01 and 2010/11, an 
increase of about 60% over and above inflation. 
Over this same period, the number of full-time 
faculty increased by 25%, from about 12,000 to 
15,000. Since student enrolment has outpaced 



the growth in the number of full-time faculty, the 
student-to-faculty ratio has increased. According 
to a recent sector publication, the average salary 
of Ontario's full-time faculty is 6% higher than the 
average for all Canadian provinces and the highest 
in Canada. 

Estimates of the ratio of full-time-equivalent 
students to full-time faculty in Ontario suggest that 
it increased from 17 students per faculty member 
in 1988 to 25 students per faculty member in 2008. 
A recent estimate prepared by the COU noted that, 
in 2008/09, if all other instructional and academic 
staff were included, the student-to-faculty ratio 
would still be about 20 students per faculty mem- 
ber, the highest of all Canadian provinces. 

An increase in the average number of students 
per faculty member typically results in an increase 
in average class size. While some studies indicate 
that factors such as the skill of the instructor and 
the teaching methods used can compensate for 
larger classes, many studies on the impact of class 
size suggest that large classes pose challenges to 
providing effective instruction, which may in turn 
impact student learning outcomes. Such challenges 
include less student interaction with faculty, and 
difficulties in maintaining the students' attention 
and motivation. In a recent Ontario Confederation 
of University Faculty Associations survey of faculty 
members at Ontario universities, 63% noted that 
average class size had increased in the last five 
years, and most respondents expressed concern 
over the increase. Those surveyed noted that the 
top three changes they had made to their teaching 
approach in response to an increased workload 
were to reduce one-on-one time with students 
outside the classroom, to give more multiple-choice 
tests and to give fewer writing assignments. 

Data we saw at two of the universities we 
visited demonstrated that their undergraduate 
class sizes had increased. Although administrators 
at one of these universities had concerns about 
the accuracy of their data, the other university 
calculated that average class size had increased 
by 20% over the last 10 years. The third university 
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had not calculated average class size but did have 
class-size distribution data for select faculties. For 
these faculties, more than 75% of available first- 
year classroom spaces were in classes of over 100 
students. In addition, we observed that, although 
the full-time-equivalent student enrolment at this 
university increased by 50% between 1998 and 
2010, the number of full-time faculty had remained 
essentially unchanged. 

Research suggests that even though larger 
class sizes have increased faculty workloads for 
each course taught, the number of courses taught 
by tenure-stream faculty has declined in the last 
20 years from a norm of 3 full courses in previ- 
ous years. At all three universities we visited, 
the expected teaching load was generally 2 full 
courses per year for tenure-stream faculty. The 
workload for a typical course consists of three 
hours of classroom instruction per week for eight 
months, as well as time spent on course prepara- 
tion, office hours to address student questions 
and time to mark tests. We were able to obtain 
sufficient information from some of the faculties 
or departments that we focused on to gain some 
insight into actual teaching loads. At two of the 
universities we visited, the actual teaching load in 
the departments reviewed that had such informa- 
tion was 1.4 courses per faculty member per year 
(although teaching-load calculations do not neces- 
sarily include the supervision of graduate students, 
which, for some professors, can be a significant 
responsibility) . At the remaining university, 
administrators estimated that the annual teach- 
ing load for tenure-stream faculty throughout the 
university averaged between 1.5 and 1.75 courses 
per year. Although the expected teaching load at 
all three universities was generally 2 full courses 
per year, there were a number of reasons why the 
actual teaching load differed, including that all 
three universities offered tenure-stream faculty 
the opportunity to take a one-year paid leave from 
the university for every six years worked, primarily 
to undertake focused research. As well, we were 
advised that the decline in course workloads over 



the last 20 years has also been due to a significant 
increase in the number of graduate students 
at some Ontario universities over this period. 
Professors who supervise graduate-level students 
typically spend less time teaching specific courses 
and more time providing one-on-one guidance or 
teaching small groups. 

The teaching load for tenure-stream faculty is 
also significantly affected by their research respon- 
sibilities. As previously noted, tenure-stream faculty 
are typically expected to devote about 40% of their 
time to research. We noted that all three universi- 
ties we visited also employed full-time or contract 
faculty considered as teaching staff, whose primary 
responsibility is to teach and who accounted for 
between about 5% and 20% of full-time-equivalent 
faculty. The expected teaching load for teaching- 
stream faculty ranged from 3 + to 4 full courses per 
year. As well, with information supplied to us by the 
universities, we calculated that the average salary 
of teaching-stream faculty for the three universi- 
ties ranged from $74,000 to $100,000 annually, 
whereas the average tenure-stream faculty salary 
ranged from $118,000 to $135,000. 

At the two universities we visited that employed 
the highest percentage of teaching-stream faculty, 
those we spoke to generally felt that using such staff 
was a good way to address resource constraints. 
However, none of the three universities visited had 
undertaken any documented analysis to determine 
whether they were using the right proportion of 
teaching-stream staff. As well, concern was raised 
at all three universities that having too many 
teaching-stream faculty on staff can affect the uni- 
versity's ability to conduct leading-edge research, 
which is generally acknowledged as being a second- 
ary, if not the primary, mandate of a university. This 
could eventually have an impact on teaching quality 
if faculty do not keep up to date. 

The available performance and promotion 
information at two of the universities suggests 
that the quality of teaching provided by teaching- 
stream faculty is as effective as that provided by 
tenure-stream faculty. A recent study on teaching 
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staff concluded that a reasonable inference can be 
made that there is a positive relationship between 
the presence of teaching faculty and the quality of 
the student learning experience. At one university, 
teaching-stream faculty had to undergo a proba- 
tionary period similar to tenure-stream faculty 
during which they were required to demonstrate 
that they were excellent teachers. Conversely 
at this same university, few faculty in the tenure 
stream had received tenure on the basis of excellent 
teaching; the vast majority of tenure awards were 
based on a combination of excellent research and 
competent teaching. 

Sessional instructors also provide an opportun- 
ity for universities to address resource constraints. 
At all three universities visited, sessional staff were 
generally paid less than $15,000 per full course 
taught. Figures provided at the three universities 
we visited show that sessional faculty accounted for 
between 10% and 24% of the full-time-equivalent 
staff. However, sessional faculty are responsible 
for a larger proportion of the teaching at all three 
universities. For example, we calculated that at the 
university where sessional faculty accounted for 
about 10% of full-time-equivalent staff, they were 
responsible for teaching 25% of the courses. 

The administrators we spoke to at all three 
universities visited indicated that there are cir- 
cumstances where sessional faculty may be the 
most qualified to teach a specific course. However, 
using these faculty is often not an appropriate 
way to address resource constraints. Among the 
reasons cited are that sessional faculty often have 
other employment commitments that take priority 
and they are usually not as available to students 
outside the classroom. At the one university that 
had institution- wide data, our analysis of student 
evaluation results identified that although students 
were generally satisfied with the performance 
of sessional faculty, teaching-stream and tenure- 
stream faculty on average tended to perform better 
than their sessional counterparts. 



RECOMMENDATION 4 



To enhance their understanding of the impact 
that use of various teaching resources has 
on teaching quality and student outcomes, 
universities should: 

• assess the impact of class size on teaching 
quality and study how best to address the 
challenges posed by large classes; and 

• weigh the impact of using teaching 
and sessional faculty and the extent to 
which they can best be used to address 
resource constraints. 




SUMMARY OF UNIVERSITIES' 
RESPONSES 



■ 



The universities agreed that teaching quality can 
have an effect on student outcomes. One univer- 
sity noted that it will undertake an assessment 
of the impact of class size on teaching quality 
and of how best to address any challenges 
posed by large classes. As well, the university 
would continue to weigh carefully the impact 
of sessional faculty and the ways in which they 
can be fairly and effectively used to help the 
university's students. Another university stated 
that it would continue to use existing research 
and data in this area to explore means to meas- 
ure the impact of various teaching resources on 
teaching quality and learning outcomes and will 
take action where opportunities are found to 
improve student outcomes. 



PERFORMANCE MEASUREMENT AND 
REPORTING BY THE MINISTRY 

In May 2011, the Ministry announced a new strategy 
for post-secondary education. The Ministry's ultim- 
ate goal is for 70% of Ontario's population to hold 
post-secondary education credentials by 2020 that 
will prepare them for the workforce of the future. 
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The Ministry expects that 70% of new jobs will 
require education and training beyond high school. 

In 2011, the Ministry reported that 64% of 
Ontarians aged 25 to 64 had post-secondary 
credentials, a notable increase from 56% in 2002. 
To further increase the number of post-secondary 
graduates, the province has pursued a growth 
strategy that has resulted in a more than 50% 
increase in the number of university students over 
the last 10 years, from approximately 250,000 full- 
time-equivalent students in 2000/01 to 390,000 in 
2010/11. In addition, as part of its May 2011 plan, 
the Ministry has committed to funding an addi- 
tional 40,000 university spaces by 2015/16. 

Working with students, faculty and post- 
secondary institutions to identify and measure the 
essential elements of teaching excellence to ensure 
that teaching is improved is another key goal of 
the Ministry's strategy. While the Ministry has 
committed significant financial resources to support 
increased enrolment in Ontario's universities, a 
significant portion of faculty surveyed recently by 
the Ontario Confederation of University Faculty 
Associations believed that the quality of under- 
graduate education had declined over the previous 
five years. The Higher Education Quality Council 
of Ontario (HEQCO) and the Commission on the 
Reform of Ontario's Public Service (Drummond 
report) have expressed similar concerns that the 
quality of education at Ontario universities has 
diminished, for reasons including increased class 
sizes and the use of more sessional instructors. The 
quality and effectiveness of teaching can have a 
considerable impact on student learning outcomes 



as well as on the Ministry's ability to achieve its 
goal to adequately prepare Ontario's workforce for 
the future. 

Although the Ministry currently does not 
directly assess teaching quality in Ontario universi- 
ties, it collects information that can assist in the 
assessment of student outcomes, which are in part 
dependent on teaching quality. For example, the 
Ministry surveys university graduates and publishes 
graduate employment rates by program type and 
institution for the periods six months and two years 
after graduation. Although it is not published, a sig- 
nificant amount of additional graduate employment 
information is also collected from these surveys, 
such as whether graduates are employed full time 
and the annual salary of graduates. 

Many university staff and administrators we 
spoke to felt that graduate employment outcomes 
can be used in combination with other indicators 
to provide insight into teaching quality. In addition, 
the Ontario Undergraduate Student Alliance as well 
as the students we spoke to at the universities we 
visited said that students want additional informa- 
tion on graduate employment outcomes to assist 
them with choosing a university and program of 
study that could help them get a good job when 
they finish university. 

The latest survey of 2008 graduates highlighted 
that more than 90% of graduates are employed two 
years after graduation. However, our calculations, 
shown in Figure I, indicate that a significantly 
smaller percentage are employed full-time in a job 
that is somewhat or closely related to the skills 
acquired in their studies. Employment outside one's 



Figure 1: Employment Situation of Ontario's 2008 University Graduation Class (%) 

Source of data: Ministry of Training, Colleges and Universities 
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original field of study may very well be rewarding 
for many graduates; nevertheless, undergraduate 
students in the process of considering their career 
options would still find it beneficial to know what 
the actual employment outcomes for graduates are. 
As well, by our calculations, only 59% of graduates 
surveyed across Ontario indicated that they were 
employed in a full-time job that required at least 
some university education. 

Other jurisdictions publish information on gradu- 
ate outcomes that is significantly more comprehen- 
sive than that published in Ontario. In particular, 
students in British Columbia are surveyed about 
their status two and five years after graduation, and 
the results are reported by survey year, discipline 
and institution. The surveys' goals are to address 
university accountability, gather relevant data for 
program evaluation and planning, and provide stu- 
dents with information to help them make informed 
decisions. In addition to employment outcomes, 
the B.C. reports include graduates' opinions on 
the usefulness of skills acquired in the program of 
study to their functioning in their jobs, the quality 
of instruction and learning outcomes. For example, 
graduates are asked their views on how their univer- 
sity program helped develop skills such as written 
and oral communication, reading and comprehen- 
sion, group collaboration and critical analysis. One 
of the universities we visited stated that it has agreed 
to participate in a new survey of graduates that is 
modelled after the B.C. survey. 

In 2005, the Ministry created the Higher 
Education Quality Council of Ontario (HEQCO) 
to generate evidence-based research to assist in 
improving the post-secondary education system. 
Since HEQCO was established, it has published 
many research papers that have a bearing on teach- 
ing quality. Topics covered include the approaches 
taken to deal with the challenges of large classes, 
the use of teaching-stream faculty to address 
increased enrolment, and the validity of student 
course evaluations as an effective measure of teach- 
ing quality. At the time of our audit, HEQCO was 



in the process of developing a set of performance 
indicators including measures to monitor, evaluate 
and improve the quality of education in Ontario's 
post-secondary education system. 



RECOMMENDATION 5 



To assist students in making informed deci- 
sions on university and program selection 
and to help achieve its goal of adequately 
preparing Ontario students for the future work- 
force, the Ministry of Training, Colleges and 
Universities should: 

• collect and make public sufficient informa- 
tion on student outcomes, including infor- 
mation on graduate employment outcomes 
and students' satisfaction with the quality of 
their education; and 

• work with the university sector to support 
the development of meaningful measures 
for student learning outcomes as a way to 
maintain teaching quality. 

| MINISTRY RESPONSE 

The Ministry supports the Auditor General's rec- 
ommendation and agrees that assessing teach- 
ing quality in the university sector is not easily 
quantifiable and is difficult to measure. 

The Ministry currently publishes student 
outcome data, including graduation and 
employment rates for university graduates, and 
it is committed to making public additional 
information that it has collected on graduate 
employment outcomes. 

As well, the Ministry is currently working 
with universities to implement an Ontario 
Education Number that will improve collec- 
tion of student-level data, and the Ministry 
will consider utilizing performance-reporting 
mechanisms already in place with universities 
to collect additional information relating to 
teaching quality. 
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As it is increasingly recognized that student 
learning outcomes may be an effective way 
to measure teaching quality the Ministry has 
made student learning outcomes a key focus of 
its future direction and has recently started to 
engage sector stakeholders, including universi- 
ties, on learning outcomes. In addition, the 
Higher Education Quality Council of Ontario 
(HEQCO) supports this focus through several 
projects aimed at defining and understanding 
student learning outcomes, including co- 
ordinating Ontario's participation in an OECD 
feasibility study, and evaluating the use of 
learning-outcome measurement tools in pilot 
universities. As well, HEQCO is working with 
the post-secondary education sector on system 
performance measures, including indicators 
of quality. 



I SUMMARY OF UNIVERSITIES' 
RESPONSES 

The universities agree that meaningful informa- 
tion would benefit students in making informed 
decisions on university and program selection, 
and they will work with the Ministry and 
the university sector to develop meaningful 
measures for student learning outcomes. One 
university noted that it already collects informa- 
tion on teaching quality and student outcomes 
and will continue to improve on these efforts. 
In 2012/13, the university will be implementing 
a Graduate Outcomes Survey for graduates of 
undergraduate programs who are five years 
post-graduation. The survey will provide valu- 
able feedback on the status, experiences and 
perspectives of graduates as they become estab- 
lished in their post-graduation careers. 



Chapter 3 
Section 




Ministry of Children and Youth Services 



Youth Justice Services 
Program 



Background 




The Ministry of Children and Youth Services (Min- 
istry) is responsible for providing community and 
custodial programs and services to Ontario youths 
aged 12-17 who have been charged with a crime 
and are awaiting trial or have been found guilty by 
the court. The Ministry also provides some services 
for youths at risk of committing a crime. The goals 
of the Youth Justice Services program are to reduce 
reoffending, contribute to community safety, and 
prevent youth crime through rehabilitative pro- 
gramming, holding youth accountable and creating 
opportunities for youths at risk. 

Less than 2% of youths in Ontario are involved 
with the youth justice system. Males account for 
75% of all cases. 

During the 2011/12 fiscal year, the average daily 
population in Ontario's youth justice system was 
about 9,200, comprising 8,600 under community 
supervision (that is, where regular reporting to a 
probation officer is generally required) and 600 in a 
youth custody/detention facility — specifically, 200 
in open custody/detention (that is, a type of group 
home in the community) and 400 in secure cus- 
tody/detention (that is, a residential setting where 
youth are restricted from leaving) . The median 
length of time in the system was one year for youths 



on probation, 60 days for youths in open custody, 
40 days for youths in secure custody, four days for 
youths in open detention and seven days for youths 
in secure detention. 

At the time of our audit, the Ministry was deliv- 
ering the program through four regional offices 
that co-ordinated and managed Youth Justice Ser- 
vices across Ontario through 64 probation offices, 
almost 500 community-based programs and servi- 
ces, 45 open custody/detention facilities operated 
by transfer-payment agencies, and 20 secure cus- 
tody/detention facilities, six of which are operated 
by the Ministry and the rest of which are operated 
by transfer-payment agencies. The facilities had a 
total capacity of approximately 1,000 beds. 

In the 2011/12 fiscal year, the Ministry spent 
$370 million on the Youth Justice Services program, 
including $168 million in transfer payments to 
approximately 200 community-based agencies. The 
federal government contributed $67 million toward 
these costs under various cost-sharing agreements. 



Audit Objective and Scope 



The objective of our audit was to assess whether 
the Ministry of Children and Youth Services (Min- 
istry) had adequate oversight and management 
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procedures in place over the Youth Justice Services 
program to ensure that: 

• legislative and judicial requirements, as well 
as program policies and procedures, are being 
complied with; 

• financial and human resources are being man- 
aged cost-effectively; and 

• the Ministry measures and reports on the 
effectiveness of its services and programs. 

Senior ministry management reviewed and 
agreed to our audit objective and associated audit 
criteria. The scope of our audit included a review 
and analysis of case files and other relevant files 
and administrative policies and procedures, as well 
as discussions with appropriate staff at the Min- 
istry's head office and at the three regional offices 
and four probation offices that we visited. We also 
visited a mix of open and secure youth custody/ 
detention facilities operated directly by the Ministry 
or by transfer-payment agencies. In addition, we 
employed a number of computer-assisted audit 
techniques to analyze youth records in relation to 
specific aspects of case management. 

We also met with staff of the Provincial 
Advocate for Children and Youth to obtain their 
perspective on the program, and we reviewed their 
latest reports on practices at two custody/detention 
facilities. We did not conduct any detailed testing 
at facilities because both the Provincial Advocate 
for Children and Youth and the Ministry's internal 
audit team had conducted reviews at some facilities 
over the past few years. 

LEGISLATIVE SCOPE LIMITATION 

In order to protect the privacy of youths, the federal 
Youth Criminal Justice Act restricts access to youth 
justice records to persons specified under the Act 
or persons designated by the Governor General of 
Canada or the Lieutenant Governor of the appropri- 
ate province. The Act does not name the Auditor 
General of Ontario as one of the people with access 
to youth justice records. As a result, in 2006, when 
conducting an audit of Children's Aid Societies, we 



requested and obtained an Order-in-Council issued 
by the Lieutenant Governor of Ontario to access 
youth files for the purpose of conducting our audit 
in accordance with the Auditor General Act. At the 
start of this audit, we confirmed with ministry rep- 
resentatives that this Order-in-Council was still in 
effect to allow us access to youth records. However, 
this Order-in-Council does not grant us access to the 
medical, psychological and psychiatric reports on 
youths resulting from court-ordered assessments. 
These reports are made available to probation offi- 
cers and can provide them with very useful informa- 
tion to support decisions on how best to rehabilitate 
youths placed under their supervision and to protect 
the public. Because we did not have access to these 
reports, we did not assess the extent to which the 
information in them was taken into consideration 
with respect to the specific programs and services 
offered to the youths who required court-ordered 
assessments of their rehabilitation needs. 




Within the past 10 years, the Youth Justice Services 
program has undergone significant transition stem- 
ming from changes in federal legislation and the 
shift in philosophy from a more incarceration-based 
approach to a community-based rehabilitation 
approach. Our best-practice research indicated that 
the trend in youth justice systems in other jurisdic- 
tions has also been to adopt a more community- 
based focus. This transition included, among other 
things, a decrease in the number of custody beds, 
an increase in community-based programs and ser- 
vices, and the separation of youths in custody from 
adults in custody. 

Between the 2005/06 and 2010/11 fiscal years, 
total program expenditures have increased by 25% 
to 30% while the total estimated number of youths 
served by the program has increased by only 5%. 
As well, ministry operating costs have grown at a 
much faster rate than funding to transfer-payment 
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agencies, even though the agencies have had to 
increase the number of programs and services 
provided due to the focus on community-based 
rehabilitation. 

Our more significant observations were as 
follows: 

• The growth in direct operating costs is due 
to a large extent to an increase in employee 
costs. Over the five-year period ending 
2010/11, all youth justice program areas, 
except for probation offices, saw a substantial 
increase in the number of full-time employees. 
Staff working in Ministry-operated secure 
facilities account for more than 60% of all 
full-time ministry staff involved with the 
Youth Justice Services program. Although 
the average daily youth population in these 
facilities decreased by 37% from 2006/07 to 
2010/11, front-line staff levels have moved 

in the opposite direction, with the number of 
full-time youth services officers increasing by 
50%. In an attempt to better understand these 
trends, the Ministry has hired consultants in 
the past few years to review cost disparities 
in open custody/detention facilities, and to 
review staff scheduling practices in Ministry- 
operated facilities. 

• In the 2010/11 fiscal year, the average bed 
utilization rate for open and secure custody/ 
detention facilities was about 50%. Over the 
years, the Ministry has made attempts to 
improve the utilization rates by reducing the 
number of beds available in the system, either 
through facility closures or by funding fewer 
beds in existing facilities. However, even with 
these efforts, the Ministry projects that, as a 
result of its community-based rehabilitation 
philosophy, the overall utilization rate will 
still be only 58% by the 2012/13 fiscal year. 

• The average daily cost per youth in custody/ 
detention facilities varies significantly by 
facility. For example, in 2011, the average 
daily cost per youth for agency-operated 
open-custody facilities ranged from $331 to 



$3,012. The average daily cost per youth at 
agency-operated secure-custody facilities 
ranged from $475 to $1,642, while those 
for ministry-operated facilities ranged from 
$1,001 to $1,483. 

• Despite low utilization rates and probable 
overstaffing in Ministry-operated secure facili- 
ties, in the 2010/11 fiscal year the Ministry 
incurred $3.9 million in overtime costs for 

all youth services officers and an additional 
$11.7 million to supplement youth services 
officers with contract staff. 

• A good initiative has been the introduction 
of the "single-case-management" model, 
where an attempt is made to have youths 
report to the same probation officer while in 
the system and if they re-enter the system. 
As well, in our review of case files, we noted 
numerous instances where the knowledge 
and experience of probation officers was put 
to good use in managing the needs of the 
youths. However, many of the required risk 
assessments and identified rehabilitation 
needs were not being documented, and many 
court-ordered conditions were either not 
being complied with or we could not deter- 
mine whether they had been complied with 
due to a lack of documentation or because 
the condition was unverifiable. 

• The caseload for probation officers dropped 
from 33 youths per officer in the 2005/06 
fiscal year to 26 per officer in 2010/11. While 
there is no written policy or guideline with 
respect to caseloads, probation managers told 
us that the historical benchmark has been 
around 30-35 cases per officer. 

• We found little or no correlation between the 
target for the number of youths to receive 
community-based programs or services and 
the amount of annual funding approved for 
service providers. We also found that the 
actual number of youths served in prior years 
had little impact on approved funding for the 
following year. 
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• The Ministry aims to deliver evidence-based 
rehabilitation programs to youths, and it 
therefore collects academic research on 
the most effective treatment models and 
approaches, which it calls the "What Works 
Literature." However, although this is a good 
initiative, we found no documentation or 
other evidence to show that programs and 
services available to youths were actually 
evidence-based or aligned with the literature. 
The Ministry assumes that transfer-payment 
agencies use the What Works Literature to 
develop evidence-based programs and servi- 
ces. But the Ministry's regional staff whom we 
spoke with and who were responsible for con- 
tracting with service providers and evaluating 
the programs and services offered were gener- 
ally not aware of the What Works Literature. 
As well, given that most programs and servi- 
ces are community-based, we expected that 
ministry staff would do site visits to assess the 
programs being offered but saw no evidence 
that they did so. We were told that regional 
program staff conduct informal evaluations 
with input from probation officers. 

• Ministry recidivism statistics exclude more 
than 80% of the youths who have come 
into contact with the Youth Justice Services 
program. The recidivism rate for youths with 
community-based sentences who are tracked 
has remained relatively stable over the past 
five years, at 35%, while that for youths with 
custody sentences who are tracked has gotten 
slightly better, decreasing to 59%. 

• In 2010, the Ministry began to pilot exit 
surveys to collect feedback from youths upon 
their release from custody/detention facili- 
ties and upon completion of their probation 
period. The Ministry expects to implement 
both types of exit surveys as a province-wide 
initiative next year. 



OVERALL MINISTRY RESPONSI 

The Ministry appreciates the work of the Aud- 
itor General and his staff and welcomes their 
input on how we can further improve youth 
justice services in Ontario. 

Since the creation of the Ministry of Children 
and Youth Services in 2004, we have trans- 
formed our service delivery model to better 
meet the needs of youth in, or at risk of, conflict 
with the law. Over the past eight years, the Min- 
istry has successfully repositioned youth justice 
services from a predominantly custody-focused 
system to one that is more community-based. 

Transformation required integration and 
realignment of two previously separate and 
distinct systems and disentanglement of ser- 
vices between ministries to build a dedicated 
youth justice system that would respond to the 
principles, provisions and sentencing options of 
the Youth CriminalJustice Act (Act). It is import- 
ant to note that the Act was introduced by the 
federal government to achieve a policy shift for 
youth justice in Canada. The intent of this shift 
was to reduce the use of custody; increase the 
use of community-based alternatives; maintain 
youth justice separate from the adult system; 
and focus on diversion, rehabilitation, reintegra- 
tion and addressing the underlying causes of 
offending behaviour by youth. 

The new service delivery model is a con- 
tinuum of community and custodial services 
that range from prevention/ diversion programs 
to providing custody and detention. The Min- 
istry uses the significant research about "What 
Works" to develop programs designed to reduce 
reoffending, address the needs and risks of spe- 
cial populations such as Aboriginal youth and 
youth with mental health issues, and provide 
gender-specific programming. The research also 
informs the development of policies, tools and 
probation case-management service delivery. 
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The Ministry's policies and processes for 
monitoring compliance and for use of resources 
have been guided by: 

• legislative requirements; 

• the shifts from custody to community experi- 
enced under the Act; 

• the responsibility to address the often 
complex needs of youth in order to improve 
outcomes for them and reduce reoffending; 
and 

• the need to maintain the safety of youth, 
staff and the community. 

A Strategic Plan is in place to guide work 
over the next two to five years in order to 
strengthen and review the gains made during 
transformation. The Ministry is committed to 
taking action in response to the Auditor Gen- 
eral's recommendations, which will enhance 
youth justice services in Ontario. 



Detailed Audit Observations 



TRANSFORMATION OFTHE YOUTH 
JUSTICE SYSTEM 

Prior to April 1, 2003, the federal Young Offend- 
ers Act governed the prosecution and sentencing 
of youths for criminal offences across Canada. 
Among other things, it outlined the requirements 
regarding custody and supervision. During this 
time, Ontario offenders under age 18 were served 
by two ministries: the Ministry of Community and 
Social Services provided services to youths aged 
12 to 15 (referred to as Phase 1), and the Ministry 
of Community Safety and Correctional Services 
provided services to youths aged 16 to 17 (referred 
to as Phase 2). In the latter case, youths in custody 
were often accommodated in designated units 
within adult correctional institutions. On April 1, 
2003, the Young Offenders Act was replaced by the 
Youth Criminal Justice Act (Act) . One of the major 



principles of this new Act was that a young person 
in custody or detention must be held separate and 
apart from adults in custody or detention. This 
caused the Ontario government to begin planning 
for a transition of youths from adult institutions 
to facilities designated as youth-only. In addition, 
the Act indicated that the appropriate level of 
custody for a young person is the one that is the 
least intrusive, after taking into consideration the 
seriousness of the offence, the youth's needs and 
circumstances, and the safety of the young person 
and society. 

On August 1, 2003, the legal and financial 
responsibility for Phase 1 youths was transferred 
from the Ministry of Community and Social Servi- 
ces to the Ministry of Community Safety and Cor- 
rectional Services. On April 1, 2004, responsibility 
for all youths was again transferred to the newly 
formed Ministry of Children and Youth Services, in 
order to clearly separate the youth justice system 
from the adult system. 

The transformation of the youth justice system 
from 2003 to 2009 involved: 

• the transfer of almost 1,400 staff from the 
Ministry of Community Safety and Correc- 
tional Services to the Ministry of Children and 
Youth Services; 

• the net reduction of 478 beds in secure 
custody/detention facilities (achieved by 
closing 429 beds in 11 adult correctional 
facilities, opening 252 beds in five newly built 
youth-only facilities and further eliminating 
301 beds due to the decline in the number of 
secure-custody court judgments); 

• the reduction of 550 beds in open custody/ 
detention facilities; and 

• a 50% increase in the number of community 
programs and services. 

Some services — such as training of custodial 
staff, nutritional services, information technology 
and statistical services — continue to be provided by 
the Ministry of Community Safety and Correctional 
Services under a service-level agreement. 
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With the introduction of the Youth Criminal 
Justice Act in 2003, the trend has been a decrease in 
the use of custody and an increase in community- 
based programs to rehabilitate offenders under 
the age of 18. Police and Crown prosecutors are 
diverting from the court process youths who have 
committed the least serious offences; for those 
youths who go through the court process, custody 
sentences tend to be reserved for the most serious 
and repeat offenders. 

THE YOUTH JUSTICE SYSTEM PROCESS 

When a youth is accused of committing an offence, 
the police will investigate the incident to determine 
whether to lay charges. The police may choose not 
to charge the youth if the offence is non- violent in 
nature and the youth has no prior convictions. In 
such circumstances, the police may use other meas- 
ures to divert the youth from the criminal justice 
system. Other measures can take many forms; for 
example, the officer may decide to take no further 
actions against the youth, give the youth a verbal 
warning or refer the youth to programs in the 
community. Although the Ministry funds some of 
these community-based programs, it does not have 
any involvement in managing the cases of youths 
diverted by police. 

If the youth is charged, the Crown prosecutor 
also has an opportunity to divert the youth from 
the criminal justice system through one of the fol- 
lowing methods: 

• referral to a Youth Justice Committee, whose 
role is, among other things, to find services 
and people in the community to provide 
supervision and mentoring; 

• referral to a youth-mental-health court worker 
who can connect youths to community-based 
mental health programs; and 

• extrajudicial sanctions, which usually require 
the youth to perform one of the following 
actions: apologize or make restitution to the 
victim, write an essay about the offending 
behaviour, donate money to a charity, perform 
community service, or attend counselling. 



The Ministry provides funding to support more 
than 30 youth-mental-health court-worker pos- 
itions. Extrajudicial sanctions are either directly 
managed by the Ministry through its probation 
offices or administered by transfer-payment agen- 
cies contracted by the Ministry. The probation 
officer or the transfer-payment agency is respon- 
sible for selecting what action is to be taken by the 
youths serving the sanctions, overseeing comple- 
tion of these actions and informing the Crown 
attorney when each case is successfully completed. 

If neither the police nor the Crown divert the 
youth from the criminal justice system, the youth, 
represented by legal counsel, is taken before a 
judge who will decide whether to hold the youth in 
detention until trial. If the court decides the youth 
is to be held until trial, the Ministry will use criteria 
outlined in the Child and Family Services Act to 
decide whether to place the youth in open deten- 
tion or secure detention. The actual facility where 
the youth is placed is based on proximity to home, 
gender and any specific risks the Ministry identifies 
during first contact with the youth, such as possible 
gang affiliations. The length of stay in detention 
can range from a few days to more than a year, 
depending on a number of factors, including the 
severity of the charges and when the trial ends. 

When requested by the court, the Ministry is 
required to provide assistance during court pro- 
ceedings, such as preparing pre-sentence reports 
for the court and co-ordinating other assessments 
regarding the youth's suitability for more intensive 
support, rehabilitation and supervision. The Min- 
istry also pays for medical and psychiatric assess- 
ments requested by the courts. 

If the youth is convicted, the court determines 
the type and length of sentence as well as any con- 
ditions applied to the sentence. The court imposes 
one of two types of sentences, which are managed 
by the Ministry: 

• Custody — The youth is placed in an open- or 
secure-custody facility for a specified period 
of time. These are the same facilities used 
to hold youths in detention until trial. All 
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custody terms are followed by a period of 
community supervision equal to one half of 
the custody period. 
• Community supervision — The youth is 
returned to the community to serve the 
sentence, which can be expressed in terms of 
days as in the case of a probation order, or in 
terms of hours as in the case of a community- 
service order. Community supervision 
typically includes regular reporting to a pro- 
bation officer and may require counselling or 
rehabilitation. 
In general, every sentenced youth is assigned to 
a probation officer to manage the youth's case while 
in the system. The probation officer is responsible 
for identifying the youth's needs, assessing his 
or her risk to reoffend, and developing a case- 
management plan to meet those needs and mitigate 
those risks. Based on the plan goals, the probation 
officer connects the youth with programs and ser- 
vices in the community, such as anger management 
counselling. Youths who are sentenced to custody 
also are assisted and monitored by youth services 
officers who work in the facilities. These officers are 
involved with the probation officers in developing 
case-management plans for youths in facilities, and 
assist the youths in carrying out the plan's program- 
ming and objectives while in custody. 

Programs and services to youth in conflict with 
the law are provided through the Ministry's Youth 
Justice Services Division. As of March 31, 2012, the 
Division, with a staff of more than 1,700 people, 
comprised a corporate office (with 117 staff), four 
regional offices (with 69 staff), 64 probation offices 
(with 512 staff) and seven secure custody/deten- 
tion facilities (with 1,039 staff). Corporate office 
provides leadership and strategic direction in the 
development of policy, procedures and program- 
ming, and is responsible for, among other things, 
staff training, investigations and financial planning. 
Regional offices co-ordinate and manage youth 
justice services across the province, including con- 
tracting with and funding transfer-payment agen- 
cies, monitoring expenditures against approved 



budgets, and conducting annual inspections of 
custody/detention facilities. Probation offices 
provide supervision and case-management services 
to most youths in Ontario convicted of a crime. 
The Ministry contracts with transfer-payment 
agencies to deliver all other services, including all 
community-based programs and services, and to 
operate the 14 remaining secure custody/detention 
facilities and all 45 open custody/detention facili- 
ties. The transfer-payment agencies, the majority of 
which are not-for-profit organizations, have been 
providing services to the youth justice program for 
a number of years. 

PROGRAM EXPENDITURES 

As a result of the shift in emphasis to a more 
community-based response for dealing with 
youths in conflict with the law, the number of 
youths in community-based programs almost 
doubled between 2005/06 and 2010/11. Over 
the same period, the number of youths placed by 
court order in custody and detention dropped by 
26%. However, contrary to what one might expect, 
this shift toward lower-cost community services 
has not brought about a decrease in total program 
costs. In fact, program costs from the 2005/06 to 
the 2010/11 fiscal years have increased for each 
type of major service activity (that is, prevention, 
diversion, agency-delivered community-based 
services and programs, probation, reintegration/ 
rehabilitation, and custody/detention) . 

Over the past several years, expenditures for 
the Youth Justice Services program have steadily 
increased, even though the estimated number 
of youths served has remained relatively stable, 
as shown in Figure 1. Between the 2005/06 and 
2010/11 fiscal years, the total estimated number 
of youths in community and custodial programs 
increased by only 5%, from about 29,000 to about 
30,500, while total program expenditures increased 
by more than 30%, from $273 million to $362 mil- 
lion. (The Ministry informed us that the $273 mil- 
lion in spending for 2005/06 is understated 
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Figure 1: Total Program Expenditures and Estimated 
Number of Youths Served, 2005/06-2010/11 

Source of data: Ministry of Children and Youth Services 
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* The Ministry tracks the number of services provided but not the number of 
individual youths being served. As youths often receive multiple services, 
counting services overstates the number served. We therefore estimated 
the number of youths served for each year by counting individuals in 
prevention programs, in diversion programs, assigned to probation officers 
and placed into custody/detention facilities. 

because it did not include the costs incurred by the 
Ministry of Community Safety and Correctional 
Services for operating youth custody/detention 
units within adult facilities, the costs of which were 
not available. In 2009/10, after the units were 
closed down, the Ministry took on the cost of all 
youth facilities, for which it received an annual 
budget increase of $18 million. While the Ministry 
had no data for the years before 2009/10, if an 
adjustment was made for this, program expendi- 
tures would still have increased 25%-30% since 
2005/06.) During the same period, federal funding 
for the Ministry's program provided under various 
cost-sharing agreements remained relatively steady 
at approximately $65 million per year. 

Between the 2005/06 and 2010/11 fiscal years, 
ministry operating costs grew proportionately more 
than funding to transfer-payment agencies (47% 
and 19%, respectively), even though the number 
of Ministry-funded programs and services offered 
by transfer-payment agencies grew by almost 40%. 
The largest increase in ministry direct operating 
expenditures was employee costs, which went up 
57%, from $94 million in 2005/06 to $148 million 



in 2010/11. Over this period, all Youth Justice Ser- 
vices program areas, except for probation offices, 
saw a substantial increase in the number of full- 
time employees, as shown in Figure 2. Staff work- 
ing in Ministry-operated secure facilities account 
for more than 60% of all full-time Ministry staff 
involved with the Youth Justice Services program. 
Although the average daily youth population in 
these facilities decreased by 37% from 2006/07 to 
2010/11, front-line staff levels have moved in the 
opposite direction with the number of full-time 
youth services officers increasing by 50%. Most 
of this increase occurred in 2008 and 2009 when 
three newly built, Ministry-operated facilities 
became operational. In the 2010/11 fiscal year, 
the Ministry operated two-thirds of all secure beds 
in the system, yet received three-quarters of total 
funding for secure facilities. The Ministry acknow- 
ledged to us that the facilities it operates directly 
are likely overstaffed at the present time and has 
started to take action that it expects will reduce 
staffing levels. 

Utilization Rates 

Custody and detention services provide residential 
care and 24-hour supervision. Understandably, this 
is the most costly service activity. The rise in the 
cost of custody/detention facilities coupled with 

Figure 2: Full-time-equivalent Ministry Staff Working 
for the Youth Justice Services Program 

Source of data: Ministry of Children and Youth Services 
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a decrease in the number of youths admitted to 
such facilities has led to lower utilization rates and 
higher per diem rates. 

"Utilization rate" is the term used to describe 
the percentage of occupied beds in a custody/ 
detention facility over a specified period of time, 
usually a year. At the start of our audit, the Ministry 
informed us that the optimal utilization rate for 
both the open and secure systems was 80%. By 
the end of our audit, the Ministry had revised this 
target to 70%. We researched utilization rates 
elsewhere and were able to find only two other 
jurisdictions that have established target utilization 
rates — the state of Virginia has established a target 
utilization rate of 80%-85% for its youth centres, 
and Wales in the United Kingdom has set a target 
utilization rate of 93%. 

Figure 3 summarizes the average daily resident 
count, daily capacity and utilization rates for secure 
and open facilities for the past five years. For the 
2010/11 fiscal year, the average utilization rate for 
open and secure facilities was about 50%. Secure 
custody/detention facilities operated by the Min- 
istry had a lower utilization rate (49%) than those 
operated by transfer-payment agencies (60%) . 

With the shift away from incarceration to more 
community-based methods of dealing with youths 
in conflict with the law, the average daily resident 
count has decreased as expected. Over the years, 
the Ministry has made attempts to improve the 



utilization rates by reducing the number of beds 
available in the system, either through facility 
closures or by funding fewer beds in facilities run 
primarily by transfer-payment agencies. However, 
the rate of bed reductions has not kept pace with 
the decreased need, particularly in secure facilities. 
For example, from the 2006/07 to 2010/11 fiscal 
years, the average daily resident count in secure 
and open custody/detention facilities has decreased 
by 26% and 9%, respectively. In comparison, the 
Ministry has decreased the number of beds avail- 
able in secure facilities by 18% and in open facilities 
by 22%. As a result, the utilization rate in secure 
facilities has gotten worse, while it has improved in 
open facilities. We noted that for secure facilities, 
most of the decrease in the average daily resident 
count has occurred in Ministry-operated facilities, 
yet bed capacity has been reduced proportionately 
more in agency-operated facilities. 

During our audit, the Ministry undertook 
another round of facility closures and bed reduc- 
tions to try to improve utilization rates. In February 
and March 2012, the Ministry closed one Ministry- 
operated secure custody/detention facility, reduced 
beds in two other Ministry-operated secure facilities 
and closed four agency-operated open facilities. 

Aside from custody/detention units that were 
closed because they were located in adult cor- 
rectional facilities, by the time we completed our 
fieldwork, the Ministry had removed in total 442 



Figure 3: Average Daily Capacity and Actual Occupancy at Youth Custody/Detention Facilities, 
2006/07-2010/11 

Source of data: Ministry of Children and Youth Services 
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secure beds from the system by closing five facilities 
and downsizing eight others, and had removed 593 
open beds from the system by closing 51 facilities 
and downsizing 11 others. The Ministry advised us 
that, in assessing closures and downsizing, it must 
take into consideration not only the average daily 
resident count but also special circumstances where 
more than the average number of beds is required. 
This may be one reason why the Ministry projects 
that in the 2012/13 fiscal year, the overall utiliza- 
tion rate will improve to only 58%, well below its 
targeted utilization rate. 

Per Diem Costs 

"Per diem" is the term used to describe the cost 
per day per youth in a custody/detention facility. 
Over the past few years, average per diem costs 
have been consistently higher for Ministry-operated 
facilities than for their agency-operated counter- 
parts. Figure 4 shows how the average per diem 
costs have increased for every type of facility from 
the 2006/07 to the 2010/11 fiscal year, but espe- 
cially for secure-custody facilities directly operated 
by the Ministry, where average per diem costs have 
increased by 86%. In contrast, average per diem 
costs increased by only 4% for agency-operated 
secure-custody facilities and by 30% for open- 
custody facilities. 

We also noted that per diem costs vary 
significantly even for similar types of facilities. 
For example, in 2011, per diem costs for agency- 
operated open-custody facilities ranged from $331 
to $3,012. Per diems at agency-operated secure- 
custody facilities ranged from $475 to $1,642, 
while those at Ministry-operated facilities ranged 
from $1,001 to $1,483. 

In 2008, the Ministry hired a consulting firm to 
conduct a cost analysis of open custody/detention 
facilities in order to identify the factors contribut- 
ing to significant cost differences across the system 
and to provide a funding method to support fair 
funding decisions. The consulting firm reported that 
disparities in per diem costs (calculated based on 



Figure 4: Average Daily Cost per Youth at Custody/ 
Detention Facilities, 2006/07-2010/11 ($) 

Source of data: Ministry of Children and Youth Services 
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capacity rather than actual utilization) were largely 
due to the different funding approaches of the for- 
mer "Phase 1" and "Phase 2" systems. Specifically, 
most former Phase 1 facilities (for youths aged 12 
to 15) had per diem costs that were greater than the 
provincial average, while most former Phase 2 facili- 
ties (for youths aged 16 and 17) had per diem costs 
that were less than the provincial average. Other 
factors that contributed to disparities included 
whether workers were unionized, facility capacity, 
residents' gender, and whether the facility offered 
gender-specific programming. The consulting firm 
identified inconsistent funding practices and large 
variances among facilities, and recommended 
a standardized approach. The consulting firm 
developed a funding tool to enable more compar- 
able funding among service providers. However, as 
noted by the consulting firm, the tool does not take 
into consideration the quality or scope of services 
being provided by the various facilities when ana- 
lyzing the appropriate funding. In response to the 
consultant's report, the Ministry increased funding 
in the 2009/10 fiscal year for those open custody/ 
detention facilities whose per diem costs at full cap- 
acity were below the provincial average, and did not 
change funding for facilities whose per diem costs at 
full capacity were above the provincial average. 
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RECOMMENDATION 1 



To help ensure that spending for secure and 
open custody/detention facilities in the Youth 
Justice Services program is commensurate with 
the need for services, the Ministry of Children 
and Youth Services should: 

• take additional steps to improve utilization 
rates by reducing bed capacity in signifi- 
cantly underused facilities; and 

• review all facilities' per diem costs for rea- 
sonableness and reduce funding for those 
whose per diem costs significantly exceed 
the provincial average, keeping in mind the 
quality and scope of services provided by 
each facility. 

| MINISTRY RESPON! 

The Ministry agrees that program expenditures 
for secure and open custody/detention facili- 
ties must be commensurate with the need for 
services. Utilization in the youth justice sector 
is determined by the degree to which courts 
make orders for detention and sentence youths 
to custody. 

The Ministry regularly monitors bed 
utilization, taking into account fundamental 
service principles such as providing services 
as close to a youth's home community as pos- 
sible, gender-dedicated services and dedicated 
services for Aboriginal youth in conflict with 
the law. In 2012/13, the Ministry will under- 
take further review of the open and secure 
custody/detention system to determine if other 
capacity reductions are possible, while respect- 
ing requirements related to maintaining safe 
environments, meeting legislative requirements 
for security, meeting the unique needs of youth 
and maintaining capacity for potential increases 
as a result of the 2012 federal amendments to 
the Youth Criminal Justice Act. 

In 2012/13, the Ministry will start a review 
of open and secure custody/detention per diem 



costs to identify the impact of underutiliza- 
tion and confirm budget reasonableness. The 
Ministry will conduct more detailed review of 
those facilities where the per diems significantly 
exceed the provincial average and make adjust- 
ments where warranted. As well, the Ministry 
will conduct a jurisdictional scan of comparable 
service providers to compare the reasonableness 
of the Ministry's per diem costs. 



FACTORS INFLUENCING EMPLOYEE 
COSTS AT MINISTRY-OPERATED 
FACILITIES 

Despite the low utilization rates and probable over- 
staffing in Ministry-operated secure facilities, the 
Ministry is also using more contract staff and incur- 
ring overtime costs. In the 2010/11 fiscal year, the 
Ministry incurred $3.9 million in overtime for all 
youth services officers, and an additional $11.7 mil- 
lion to supplement youth services officers with 
contract staff. According to an external review of 
staffing costs at directly operated facilities in 2011, 
costs of contract staff and overtime were incurred 
because of absenteeism, medical accommoda- 
tions, training, vacancies, scheduling practices and 
deployment practices. We reviewed some of these 
causes to determine if the Ministry had taken any 
mitigating actions to reduce the associated costs. 

Sick Leave 

In 2008, youth services officers at custody/deten- 
tion facilities took an average of 21 sick days, about 
two times the provincial average sick leave for gov- 
ernment employees for that year. In our last audit 
of adult correctional facilities, we found that this 
issue is also common among corrections officers at 
those facilities. To reduce absenteeism across youth 
custody/detention facilities and adult correctional 
facilities, the Ontario government implemented the 
following initiatives: 
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In August 2009, it introduced the Attendance 
Support and Management Pilot Program 
for youth services officers, youth probation 
officers, corrections officers, adult probation 
officers and their managers. After a certain 
number of absences, a meeting is triggered 
with the employee to discuss his or her 
attendance record and offer assistance such 
as information about the Employee Assistance 
Program. A progress report issued by the Min- 
istry of Government Services in September 
2011 provided results in aggregate. We calcu- 
lated that 45% of youth services officers and 
19% of youth probation officers were required 
to participate in the program between August 
2009 and June 2011. The Ministry did not 
have information on the number of employees 
who successfully exited the program, but the 
Ministry of Government Services reported 
that for the adult and youth systems com- 
bined, two-thirds of employees who entered 
the program successfully reduced their sick 
days to below seven days for one year. 
The government negotiated a bonus payout to 
all youth services officers and corrections offi- 
cers beginning March 2009 if their combined 
provincial sick time average was less than or 
equal to the targets set out in the Ontario Pub- 
lic Service Employees Union Collective Agree- 
ment, which in 2011 was 20 days. Since this 
initiative was introduced, actual sick days for 
youth services officers dropped from 21 to 14 
in the past three years. The Ministry has paid 
out a total of $1.5 million in bonuses from 
this initiative. At our request, the Ministry of 
Government Services estimated the cumula- 
tive net savings to be $4.5 million, but we 
could not confirm whether this estimate was 
reasonable because of conflicting information 
on sick days obtained from the Ministry of 
Children and Youth Services and the Ministry 
of Government Services. 

The external review of employee costs at 
Ministry-operated custody/detention facilities 



stated that although the initiative has good 
intentions, as it currently works it rewards 
both those misusing and those not abusing 
sick-time credits. The review also indicated 
that there seemed to be a sense of entitlement 
among staff at many of the Ministry-operated 
facilities with regard to the use of sick time. 

Medical Accommodations 

When an employee is temporarily or permanently 
unable to perform the essential duties of his or her 
job because of a medical condition or disability, 
the Ministry is required by a government-wide 
directive to accommodate the worker in another 
position that does not demand the same physical 
abilities. For example, a youth services officer hired 
as a front-line worker with youths in a facility who 
can no longer perform his or her duties might be 
put into an administrative position. We found that 
in the 2010/11 fiscal year, 6.5% of youth services 
officers in Ministry-operated facilities were on long- 
term permanent accommodation. One facility had 
as much as 17.5% of full-time youth services officers 
in such temporary or permanent accommodation 
positions. This occurred more frequently at facili- 
ties that have been in operation for some time. 

Staff Scheduling Practices 

Custody/detention facilities are staffed based on 
100% capacity. Although contract staff are to be 
used on an as-needed basis, we understand that 
many Ministry-operated facilities fill vacancies 
with contract staff without evaluating the needs of 
the site on that particular day — that is, taking into 
consideration the occupancy level. This leads to 
overstaffing, since on average only half the beds in 
a facility are occupied. 

Furthermore, we noted that even if all the beds 
were filled, the facilities might still be overstaffed. 
Under the Child and Family Services Act (Act), all 
custody/detention facilities are required to main- 
tain a minimum staff -to-youth-in-custody ratio of 
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1:8 — that is, one youth services officer for every 
eight youths in a facility In this context, "staff 
refers to youth services officers only and excludes 
facility management, administrative staff and other 
staff with specialized duties such as kitchen staff. 
Although operators of custody/detention facilities 
prepare staffing schedules, the Ministry does not 
monitor staff-to-youth ratios even for the facilities 
it operates directly During one of our visits to a 
Ministry-operated facility, we noted the facility was 
routinely overstaffed in comparison to the legisla- 
tion, as the staff-to-bed ratio in the residential units 
was 1:3 at peak hours (4 p.m.- 9 p.m.) and 1:5 at 
non-peak hours (overnight) . We were also told at 
this facility that there were occasions when some 
youths who were excessively disruptive received 
one-to-one and two-to-one supervision to de- 
escalate situations. These staffing situations would 
not be captured in the ratio above. 

At one of the facilities we visited, we observed 
other staff scheduling practices that result in high 
employee costs: 

• Both the infirmary and the area with the isola- 
tion rooms are staffed with at least one youth 
services officer at all times, whether or not 
they are occupied. 

• Youths who are suicidal require constant 
two-on-one supervision. Ideally, such youths 
should be in a facility that specializes in men- 
tal health issues or in a psychiatric hospital. 
Currently only one facility in the youth justice 
system is accredited as a children's mental 
health centre, and it often does not have the 
capacity to meet the demand for its services. 
For example, we noted that in 2011, about 265 
youths were identified as suicidal, and only 
18 spent any length of time in this specialized 
facility. 

Use of Secure Isolation 

Provisions under the Act allow a secure custody/ 
detention facility to place a youth in secure isola- 
tion if that youth's conduct indicates that in the 



immediate future the youth is likely to cause serious 
property damage or serious bodily harm to another 
person, and no less restrictive method of managing 
the youth's behaviour is practical. A youth services 
officer who is assigned to watch a youth in secure 
isolation is not available to supervise others in the 
facility. We found that Ministry-operated facilities 
make use of secure isolation rooms more often than 
agency-operated facilities. For example, in 2011 
more than 90% of the placements in secure isola- 
tion occurred in a Ministry-operated facility, and 
the average length of stay in secure isolation was 25 
hours in a Ministry-operated facility compared to 
two hours in an agency-operated facility. 

The Ministry indicated that one of the reasons 
secure isolation was used more in Ministry-operated 
facilities is that suspected gang members are typ- 
ically accommodated in ministry facilities and that 
these youths are more frequently engaged in behav- 
iour that creates an imminent risk to others, which 
is one of the criteria for using secure isolation. But 
we noted that the percentage of youths residing in 
secure facilities who were suspected of belonging to 
gangs as of December 31, 2011, was comparable — 
that is, 15% of youths in Ministry-operated facilities 
compared to 12% of youths in agency-operated 
facilities. Furthermore, we noted that when one 
Ministry-operated facility, which routinely used 
secure isolation, was closed in March 2012 and its 
youths were sent to an agency-operated facility, the 
agency-operated facility reported no use of the isola- 
tion room in the following month. 



RECOMMENDATION 2 



To help reduce employee operating costs, 
particularly at Ministry-operated facilities, the 
Ministry of Children and Youth Services should: 
• staff custody/detention facilities on the basis 
of expected utilization and not on the basis 
of full capacity, and use contract staff to fill 
vacancies only after evaluating the short- 
term staffing needs of the site; 
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• reassess whether the bonus payouts from the 
sick-day program are proving to be a cost- 
effective strategy in reducing absenteeism; 

• investigate high rates of long-term perma- 
nent medical accommodation and, where 
appropriate, implement measures to reduce 
those rates; and 

• identify behaviour-management techniques 
other than secure isolation that have been 
used successfully by agency-operated 
facilities to prevent or manage undesirable 
behaviour. 

| MINISTRY RESPON! 

The Ministry acknowledges the importance of 
effective staffing practices at its directly operated 
facilities. Staffing in custody/detention facilities 
must be done to meet the minimum staff ratio 
under the Child and Family Services Act; the 
requirement for 24/7 intake; ability to maintain 
the buildings and provide key services (for 
example, programming, education, health, clin- 
ical, case management) regardless of how many 
youth are present; as well as safety and security 
requirements. The Ministry is in the process of 
implementing the recommendations resulting 
from the review of staff deployment and sched- 
uling practices undertaken in 2011. In addition, 
a scheduling application is under development, 
which will result in more consistent and effective 
staff scheduling practices, leading to decreased 
staff costs. The Ministry will continue to evaluate 
capacity and take action where possible, while 
addressing operational requirements, meeting 
the complex needs of youth in the facilities, and 
having due regard for the safety for youth, staff 
and the community. 

Over three years, the government has 
made substantial and measurable progress 
toward reducing absenteeism and its associated 
costs, and sustaining this improvement across 
the Ontario Public Service Employees Union 
(OPSEU) Correctional Bargaining Unit. The 



Ministry of Government Services will reassess 
the program, the policy and the incentive pay- 
ments as it prepares for collective bargaining 
with OPSEU. The Ministry of Children and 
Youth Services will work with the Ministry of 
Government Services in this regard. 

The Ontario Public Service has recently 
initiated a disability management review pro- 
cess with the goal of implementing best prac- 
tices to enhance employment accommodation 
and return-to-work performance and outcomes. 
A component of the review involved completing 
an assessment of current practices and perform- 
ance relating to disability management policy 
and workplace resources, disability prevention, 
and early intervention and return-to-work pro- 
cesses. As a result of this review, an action plan 
will be developed to implement enterprise-wide 
program improvements. The Ministry will work 
with the Ministry of Government Services in 
this regard. 

We acknowledge the Auditor's concern relat- 
ing to the use of secure isolation. The Ministry is 
undertaking a review of the use of secure isola- 
tion in directly operated facilities to measure 
operational compliance with legislation, poli- 
cies and procedures. The Ministry has in place 
training, a relationship custody approach, tools, 
policies and procedures relating to behaviour- 
management techniques and the management 
of aggressive behaviour by youth. Throughout 
2012-14, the Ministry will undertake reviews 
of directly operated youth centres using the 
Correctional Program Assessment Inventory, 
which, among other things, is used to review 
the types of behaviour-management strategies 
used, whether the strategies are aligned with 
the research and whether there are documented 
procedures for applying the strategies. In 
addition, the Ministry will identify behaviour- 
management techniques that have been used by 
agency-operated facilities to determine whether 
their use is appropriate for its own facilities. 
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CASE MANAGEMENT 

Case management is the cornerstone to helping 
a youth become a law-abiding citizen, and there- 
fore is core to contributing to community safety 
and prevention of youth crime/reoffending. The 
Ministry's objective in case management is to take 
maximum advantage of the time that a young per- 
son is in the youth justice system, using that time 
to change his or her mindset and/or other aspects 
of his or her situation that could reduce the likeli- 
hood of reoffending. 

The time frame in which the youth is under the 
care of the Ministry and its service providers is lim- 
ited. In the 2011/12 fiscal year, the median length 
of time a youth was in the system was one year for 
a youth on probation, 40 days for a youth in secure 
custody, 60 days for a youth in open custody, seven 
days for a youth in secure detention and four days 
for a youth in open detention. Therefore, making 
the most of this limited time is critical to realizing 
the changes that the Ministry intends case manage- 
ment to achieve. 

Assessing Youth and Planning Services 

The probation officer conducts a risk/needs assess- 
ment to assess the youth's risk of reoffending and 
determine the level of supervision needed. The pro- 
bation officer considers the youth's history, lifestyle 
and personal circumstances. As well, we were told 
that the probation officer considers information 
contained in psychological and psychiatric reports 
resulting from court-ordered assessments. A youth 
is rated as high, medium or low risk in eight areas: 
prior offences; family circumstances/parenting; 
education; peer relations; substance abuse; leisure 
and recreation; personality; and attitudes/orienta- 
tion. The results of the assessment are then used to 
decide on a supervision level and to develop a case- 
management plan. 

In practice, the risk/needs assessment and the 
case-management plan are one consolidated docu- 
ment that the probation officer completes in collab- 



oration with the youth and other stakeholders — the 
youth's parents, for example, or the primary youth 
services officer if the youth is in custody. The plan 
details how the risks identified in the assessment are 
to be addressed, and typically specifies one or more 
programs that the youth must attend, how often the 
youth must report to the probation officer, and how 
to comply with the conditions of the court order. 

If a youth is in custody, facilities must also 
prepare a case-management reintegration plan 
for him or her within 30 days of admission, and 
update it 90 days after admission, six months after 
admission and every six months thereafter. The 
reintegration plan is intended to complement the 
case-management plan, and details what action 
should be taken during the custody phase. 

We reviewed a sample of case files closed in 
2011 for a variety of sentence types. Overall, we 
found the system to be flexible and that the know- 
ledge and experience of probation officers was 
being put to good use. For example, although the 
risk/needs assessment automatically calculates a 
risk score for the youth, we found instances where 
the probation officer overrode the score on the 
basis of personal observations and knowledge of 
the particular rehabilitation needs of the youth. 

However, we did identify some concerns with 
the risk/needs assessment process and case- 
management planning: 

• One-third of the risk/needs assessments and 
case-management plans we reviewed were 
completed late, on average by almost 60 days, 
while another third were never done, even 
though they are required by policy given the 
length of the youths' sentences. Overall, only 
60% of the files reviewed contained all the 
required risk/needs assessments. 

• Most case-management plans were incom- 
plete. They failed to include all required items, 
such as specific timelines for monitoring and 
referrals, or they did not indicate the manner 
in which compliance with court-ordered con- 
ditions would be monitored. 
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• Although most case-management plans 
included goals to address at least some of the 
youth's risk factors to reoffend, areas assessed 
as high risk were often not addressed. Overall, 
more than half of the files we reviewed for 
youths with completed case-management 
plans did not have goals for at least one of 
their high-risk factors. More than 20% of 
cases did not have goals for any of their high- 
risk factors. 

• We found that reintegration plans were gener- 
ally prepared on a timely basis. Reintegration 
plans are meant to outline specific goals for 
the youth to work on while in custody. How- 
ever, the goals in some plans were vague — for 
example, "youth should better himself by 
attending program or counselling as rec- 
ommended by clinical staff and probation 
officer," rather than specifying the particular 
in-house rehabilitation service or program 
that would benefit the youth. 

Monitoring Efforts by Probation Officers 

A probation officer is the first line of contact for 
most sentenced youths throughout the time the 
youths are in the care of the Ministry. Probation 
officers are responsible for monitoring the youth's 
progress in meeting the goals set out in the case- 
management plan and making referrals to programs 
and services based on the conditions set out in the 
court order, the case-management plan, and other 
needs of the youth identified during the supervision 
period. Probation officers must also monitor compli- 
ance with all conditions set out in the court order 
in accordance with the Youth Criminal Justice Act. 
When a youth does not comply, probation officers 
are required under ministry policy to consider 
appropriate enforcement options and to document 
their decision. Enforcement options can include 
issuing a warning, increasing the level of super- 
vision, asking the court to change the conditions of 
the court order, bringing the youth back to court on 



charges stemming from the terms of the court order 
he or she has breached, and taking no action. 

The Ministry has a "single-case-management" 
policy, in which a youth remains under the super- 
vision of the same probation officer for the duration 
of his or her court order, and any youth who comes 
back into the system is reassigned to that same pro- 
bation officer so long as the reassignment continues 
to serve the youth's best interest. This relationship- 
building model is meant to emphasize the quality of 
interaction between the probation officer and the 
youth; the importance of creating stable, consistent 
care; and the value of developing individualized, 
meaningful case-management plans and interven- 
tions. We felt that this was a good policy but noted 
that the Ministry does not monitor whether its 
single-case-management policy is being followed, 
particularly for youths who re-enter the system. In 
addition, for the youths in our sample, which rep- 
resents a single continuous period of supervision, 
more than 40% were under the supervision of more 
than one probation officer, while 6% saw at least 
five probation officers during their period of super- 
vision. In most cases, we were unable to determine 
the reasons for the reassignments because they 
were not documented in the file. 

Our review of the work of probation officers 
found the following: 

• For a significant number of files, there was no 
evidence that probation officers had made a 
sufficient effort to assist youths to meet at least 
some of their established goals. For example, 
where a youth's goal was to abstain from 
substance abuse, the probation officer made 
no referral to a substance-abuse support group 
or counselling. In another case, the youth 
had a goal to complete four credits at school, 
but there was no evidence in the file that the 
probation officer had worked with the youth 
on the issue of school attendance. In both 
examples, the youths did not meet their goals. 

• Youths are required to meet regularly with 
their probation officers, but in almost half of 
the case files we looked at, we found youths 
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had not reported to their probation officers as 
frequently as required given their risk level. 
This was due to probation officers failing to 
schedule appointments as frequently as they 
should have, or to follow up on appointments 
that the youths missed. 
Thirty percent of the community-service 
orders were not complied with on a timely 
basis. As well, we could not verify compliance 
for an additional 7% of community-service 
orders because there was no documentation 
in the case file. 

Less than 40% of conditions in the files 
we examined were complied with. For the 
remainder, almost 20% were not met and 40% 
could not be verified, either because there was 
a lack of documentation in the file or because 
the conditions set out in the orders were 
unverifiable — for example, "keep the peace," 
"don't possess a firearm," or "don't associate 
with person X." 

When a probation officer is aware that a 
court order has been breached, it is his or her 
responsibility to take appropriate action. We 
would expect this action to be in the form of 
an escalation of consequences, depending on 
the circumstances, starting with a warning 
and ending by bringing the youth back to 
court on charges of breaching the terms of the 
court order. However, in more than a quarter 
of the cases sampled where the court order 
was breached, the probation officer did not 
take timely and/or appropriate action through 
progressive enforcement. The Ministry indi- 
cated that the courts discourage probation 
officers from bringing forward breaches that 
might not be considered serious in nature, 
and that the courts expect probation officers 
to find ways of enforcing conditions without 
resorting to returning the youth to court. 
For youths who have committed crimes that 
are not serious in nature, the Crown may 
divert them from the court system if the 
youths agree to participate in extrajudicial 



sanctions. We found that there are no Ministry 
guidelines on the type of sanction to be used 
for the type of offence or type of history — for 
example, if the youth is a repeat offender. 
Consequently, sanctions employed varied by 
case and by probation officer. For example, 
for the same offence one youth was assigned 
community-service hours while another only 
had to write a letter of apology. Although these 
might well have been appropriate sanctions for 
each youth given the individual circumstances, 
supervisory staff have no way of evaluating the 
appropriateness of the sanction because none 
of the files sampled contained the underlying 
rationale for the sanction chosen. 

Probation Officer Caseloads 

Each probation officer takes responsibility for a 
certain number of youths and manages them dur- 
ing their stay in the youth justice system. Although 
there is no documented caseload standard for 
probation officers, probation managers we met 
with indicated that a benchmark of 30-35 cases per 
probation officer was considered reasonable. 

According to the Ministry's analysis, the case- 
load for probation officers dropped from 33 youths 
per officer in the 2005/06 fiscal year to 26 youths 
per officer in 2010/11. The Ministry told us that it 
responded to this decrease by shifting the super- 
vision of community-service orders from transfer- 
payment agencies to probation officers in 2011. 

Ensuring Quality of Case Management 

Probation managers are required to review case 
files annually to monitor probation officers' compli- 
ance with standards, policy and procedures. The 
Ministry has developed a checklist — with more 
than 100 standards on case-file documentation 
and monitoring — for probation managers to use to 
assist them with this oversight responsibility. We 
attempted to evaluate the case-management mon- 
itoring process at four probation offices in the three 
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regions we visited, but one probation office did not 
maintain any documentation for the three years 
we chose to examine, and another office could not 
locate any of the reviews for the 2010/11 fiscal year. 
In general, we found a variety of practices being 
used across offices, as follows: 

• Ministry policy requires at least five active 
files and two closed files to be reviewed per 
probation officer annually. However, over 
the past three years, we found that generally 
only three case-file reviews per probation 
office were being done, and in some cases as 
few as one. This represents about 12% of the 
average probation officer caseload of 26. We 
could not confirm that every probation officer 
was reviewed because probation offices did 
not maintain lists of probation officers who 
worked in that office for the years we tested. 

• There was inconsistency among probation 
offices on how files were selected for review. 
For example, in one region probation officers 
were allowed to select most of their files them- 
selves for managers to review. In the other 
two regions, probation managers selected files 
randomly from the case register. 

• Each case file is reviewed against more than 
100 standards. The probation manager judges 
whether each standard has been met, partially 
met or not met. For a sample of case-file 
reviews conducted in the 2011/12 fiscal year 
that we examined, we noted that almost two- 
thirds met at least 80% of applicable standards. 

• None of the regions we visited had analyzed 
the results of case-file reviews to identify which 
standards probation officers consistently had 
the most difficulty complying with, so that this 
information could be used in staff training. 
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To help ensure that case-management efforts 
result in youths obtaining the services and pro- 
grams needed for rehabilitation, the Ministry of 
Children and Youth Services should: 



• complete all required risk/needs assess- 
ments, case-management plans and case- 
management reintegration plans on a timely 
basis; 

• ensure that case-management plans have 
specific goals and recommended programs 
and services to assist youth in addressing 
all high-risk areas identified and any court- 
ordered conditions; 

• clearly document in the case files whether or 
not youths have complied with court-ordered 
conditions and community-service require- 
ments and, if they have not, what efforts were 
made by the probation officer to rectify this; 

• develop guidelines or policies about what 
types of extrajudicial sanctions are appropri- 
ate to use and when; and 

• ensure that the required case-file reviews are 
being done consistently across all probation 
offices and determine whether there are any 
systemic issues warranting additional guid- 
ance or training. 



| MINISTRY RESP0N 




The Ministry is pleased that the Auditor 
acknowledged the knowledge and experience of 
probation officers, and the actions already taken 
to address the recommendations. The Ministry 
launched a Probation Strategy in May 2012 that 
will strengthen probation services, account- 
ability and compliance. In addition, the Ministry 
will review its processes and implement new 
procedures where necessary to: 

• ensure that files are fully documented and 
clearly demonstrate that appropriate action 
was taken on a timely basis; 

• support quality assurance and effective use 
of Risk/Need Assessments (RNAs) by proba- 
tion officers; and 

• ensure that all required risk/need assess- 
ments, case-management plans and 
case-management reintegration plans are 
completed on a timely basis. 
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In order to track the number of outstanding 
RNAs, the Ministry is developing a database 
report that identifies "future due" and "overdue" 
RNAs so that timely action can be taken by pro- 
bation managers. 

The Ministry agrees that it is important 
for the case-management plans to address all 
high-risk areas and will review the practices 
and training that support probation officers to 
increase their capacity to engage youth to set 
and achieve goals on their own. 

The Ministry will review its processes and 
implement new procedures to ensure that files 
are fully documented and clearly demonstrate 
that appropriate actions were taken on a timely 
basis, particularly in respect of court-ordered 
conditions. 

The Ministry will review and revise, if 
necessary, its existing policies for extrajudicial 
sanctions to ensure they support selection of 
appropriate extrajudicial sanctions. 

The Ministry recognizes the importance of 
completing case-file reviews consistently. In 
October 2011, the Ministry released an updated 
Case Management Compliance Review Tool and 
Managers' Guide, which articulates expectations 
and provides guidance on conducting case-file 
reviews. In 2013/14, the Ministry will initiate 
a review of the case-file review process and 
will specifically look at accountability and con- 
sistency in case-file reviews, as recommended 
by the Auditor. The Ministry will also assess 
whether there are any systemic issues that war- 
rant additional guidance or training. 



PROGRAMS AND SERVICES 

The Ministry contracts with transfer-payment agen- 
cies to deliver a variety of programs and services, 
primarily directed to youths who come into conflict 
with the law, although they also deliver some 
preventive programming. As seen in Figure 5, these 



programs and services are grouped into six major 
service activities, and each offers categories of pro- 
grams and services. For example, community-based 
programs/services include attendance centres and 
managing youth with community-service orders. 
Reintegration and rehabilitation includes employ- 
ment- and life-skills training, anger management 
counselling, substance abuse counselling and hous- 
ing support. The Ministry contracts with almost 
200 transfer-payment agencies to offer almost 
500 services and programs across the province. 
The type of programming and services youths will 
receive is determined by their risk/needs assess- 
ment, and sometimes is listed on their court order. 

Transfer-payment agencies are funded on a 
historical basis. One-time funding adjustments are 
permitted for non-recurring expenses. Permanent 
adjustments are permitted when there is a change 
in the services or programs provided. 

Effectiveness of Agency Programs and 
Services 

The Ministry aims to deliver evidence-based pro- 
grams to youths in conflict with the law in order to 
rehabilitate them. To this end, it collects academic 
research on the most effective treatment models 
and approaches, which it calls the "What Works 
Literature." However, there was no evidence to show 
that the programs and services available to youths 
actually were selected based on the best practices 
outlined in the literature. The Ministry's corpor- 
ate office told us that it assumes transfer-payment 
agencies use the What Works Literature to develop 
evidence-based programs and services. But the Min- 
istry's regional staff to whom we spoke, who were 
responsible for contracting with service providers 
and evaluating the programs and services offered, 
were generally not aware of the What Works Litera- 
ture. Consequently, no one ascertains whether any 
of the programs and services developed by transfer- 
payment agencies align with the best treatment 
approaches according to the research. 
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Figure 5: Funding and Service Activity, 2010/11 

Source of data: Ministry of Children and Youth Services 



Funding 



Youth Served/ 
Admissions Programs/Services 



1 Service Activity 


($ million) 


% 


# 


% 


# % 


Prevention 


2.9 


1 


3,637 


8 


51 


10 


Diversion 


6.5 


2 


10,386 


21 


94 


19 


Community-based programs/services 


24.8 


9 


9,011 


18 


112 


23 


Reintegration/rehabilitation 


12.5 


4 


9,395 


19 


102 


21 


Probation 


50.1 


18 


8,673 


18 


66 a 


13 


Custody/detention 


191.2 


66 


7,927" 


16 


70 c 


14 


Total 


288.0 


100 


49,029" 


100 


495 


100 



a. Includes 64 Ministry-operated and two agency-operated probation offices. 

b. Includes 6,819 admitted to detention and 1,108 admitted to custody. 

c. Includes seven Ministry-operated and 63 agency-operated facilities. 

d. Some youths have been counted more than once if they received services from more than one service provider. 



The Ministry does not have a master list of all of 
the youth justice programs and services offered by 
the approximately 200 transfer-payment agencies 
across the province. Services are detailed in indi- 
vidual contracts with agencies, but these details are 
not catalogued in regional inventories, either. As a 
result, probation officers, who are responsible for 
connecting youths with programs and services that 
best meet their assessed needs and risk level, must 
rely on their own experience and on informal dis- 
cussions with other probation officers and contacts 
in the community to determine which programs 
and services are available in their region. Although 
this type of informal interaction among colleagues 
and contacts is undoubtedly valuable, without good 
information on the specific programs and services 
available in each region, there is a risk of inequities 
across regions and a risk that youths might not be 
connected with the services and programs that best 
meet their needs. 

Comparing and evaluating services and pro- 
gramming was also complicated by inconsistent 
terminology in the Ministry's contracts with 
transfer-payment agencies. Some used different 
wording for what seemed to be the same service. For 
example, "job readiness," "job preparedness" and 
"employment search" were used interchangeably, 



as were "education advocacy," "school attendance" 
and "school support." We noted that one agency 
just grouped all services under the umbrella term 
"youth support program," providing little informa- 
tion on the actual services it offered. It was difficult 
to determine from the contracts specifically what 
rehabilitation service and/or program was actually 
being offered by each agency. 



RECOMMENDATION 4 



To ensure that effective programs and services 
are offered to youths no matter where they live 
in Ontario, the Ministry of Children and Youth 
Services should: 

• ascertain that the services and programs 
contracted for actually align with best- 
practice youth rehabilitation research; and 

• establish and maintain a master list of 
regional programs and services that uses 
consistent terminology and make this infor- 
mation available to all probation officers. 



INISTRY RESPONSE 



The Ministry agrees that effective and evidence- 
informed programs and services must be offered 
throughout Ontario. The Ministry's service 
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continuum, which provides a range of commun- 
ity and custodial programs and services across 
the province, was guided by the "What Works" 
literature to promote evidence-based program- 
ming responsive to the needs of youths and to 
contribute to community safety. This approach 
aligns with the Youth Criminal Justice Act focus 
on addressing the causes of offending behaviour 
and promoting rehabilitation and reintegration. 

A framework for program evaluation is 
under development to confirm the Ministry's 
expectations for services and determine 
whether the various programs adhere to the 
intended design elements. 

The Ministry has completed an inventory of 
programs for its directly operated youth centres 
and is currently undertaking an inventory of 
programs in open and secure transfer-payment 
facilities. An inventory of programs and ser- 
vices offered in community agencies will be 
developed over the coming year and will be 
made available for use by all service providers, 
including probation officers. 



Funding and Monitoring of Programs and 
Services Offered in the Community 

The Ministry enters into annual service contracts 
with each of its transfer-payment agencies for ser- 
vices offered to the Youth Justice Services program. 
Among other things, the contract outlines a descrip- 
tion of the services to be provided, the amount of 
annual funding, and service targets to be achieved. 

We identified the following with respect to the 
Ministry's funding and monitoring of third -party 
agencies: 

• We found little or no correlation between 
service-level targets and the amount of annual 
funding the Ministry approved. About 60% 
of services and programs that had a funding 
increase of at least 10% in the 2011/12 fiscal 
year did not have a corresponding increase in 



the agencies' target for number of youths to be 
served. We also found agencies with similar 
service-level targets for the same service 
being funded at different levels. For example, 
in 2011/12, the Ministry approved a budget 
of $9,000 for one agency and $32,000 for 
another to each manage community-service 
orders for 25 youths. In another example, 
the Ministry approved a budget of $107,000 
for one agency and $165,000 for another to 
each manage 300 youths serving extrajudicial 
sanctions. As a result, we found significant 
variations in actual program costs per youth, 
as shown in Figure 6. 

• An agency's performance has little impact on 
the funding it will receive the following year. 
For example, in the 2010/11 fiscal year more 
than half of the programs and services pro- 
vided in the community by agencies did not 
meet their targets for the number of youths 
served, but almost 70% of these received 
the same funding amount or more for the 
following year. The Ministry told us that the 
agencies have no control over the number of 
youths served, and given the fact that agen- 
cies have mostly fixed costs, reducing funding 
would likely mean staff layoffs. 

• Budgets are adjusted throughout the fiscal 
year, some as late as March 31, resulting in 
little variance between the approved budgeted 
amounts and actual expenditures at year-end. 
Most funding adjustments happen in the 
fourth quarter. During the 2010/11 fiscal year, 
budgets were adjusted during the year for 

Figure 6: Actual Costs per Youth for a Sample of 
Community-based Programs and Services, 
2011/12 ($) 

Source of data: Ministry of Children and Youth Services 



1 Program/Service 


Lowest 


Highest 


Median 


Attendance centre 


989 


13,037 


5,208 


Anger management 


256 


9,704 


1,437 


Substance abuse 


321 


8,114 


1,264 
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half of all contracts, and 85% of the funding 
adjustments were increases. We selected a 
sample of adjustments in 2010/11 from each 
of the three regions we visited and noted that 
50% of one-time adjustments had little sup- 
porting documentation from the agency to 
substantiate the need for additional funding. 
The Transfer Payment Accountability directive 
requires ministries to establish risk criteria 
to assess the ability of service providers to 
meet service-delivery objectives. To this end, 
the Ministry has developed a risk-assessment 
questionnaire to be completed by the service 
providers. The Ministry uses the agencies' 
self-assessments to determine the overall risk 
level. Of the agencies that completed assess- 
ments in the 2011/12 fiscal year, other than 
those providing custodial services, almost all 
assessed themselves as low risk, yet most did 
not meet their service-level targets for more 
than 50% of services and programs they were 
contracted to provide. 

A report issued by the U.S. Center for Juvenile 
Justice Reform in December 2010 states that 
rehabilitative programs, if implemented well, 
can reduce recidivism substantially. However, 
there was no documented evidence that the 
Ministry or the service providers assessed the 
effectiveness or quality of the programs and 
services offered. The Ministry performs only 
limited analysis on service providers. On an 
annual basis, the Ministry compares service- 
level targets with the actual number of youths 
served. We were told that regional program 
staff conduct informal evaluations with input 
from probation officers. Furthermore, aside 
from visits to custody/detention facilities, 
there was no evidence of site visits by Min- 
istry staff to community-based services and 
programs. In accordance with its contract, the 
service provider outlines how it will evaluate 
each program it offers. We inquired whether 
the Ministry followed up with agencies to 
review the nature and results of evaluation. 



Program supervisory staff at each of the three 
regional offices we visited told us that they do 
not routinely ask for evidence of evaluations 
done. One region informed us that only one 
agency submitted an annual quality assur- 
ance report that detailed the evaluation it 
undertook, but even in this case, the agency 
had not performed the detailed analysis it had 
committed to in its contract. 



RECOMMENDATION 5 



To ensure that funding provided to transfer- 
payment agencies is commensurate with the 
value of services provided, the Ministry of Chil- 
dren and Youth Services should: 

• ensure that approved funding to agencies is 
appropriate for the expected level of service, 
based on levels of service achieved in the last 
few years; 

• compare and analyze agency costs of similar 
programs across the province, and investi- 
gate significant variances that seem unjusti- 
fied; and 

• ensure that requests for additional funding 
are adequately supported. 

| MINISTRY RESPONSE 

The Ministry agrees that its transfer-payment 
partners in service delivery should demonstrate 
value for services provided. The Ministry will 
conduct a review of Youth Justice Services 
targets, youths served and related costs over a 
three-year period to assess whether funding is 
appropriate and will make adjustments where 
warranted. In addition, the Ministry will conduct 
a jurisdictional scan of comparable Youth Justice 
Services providers to compare the reasonable- 
ness of the Ministry's transfer-payment costs. 

Starting with the 2013/14 budget year, the 
Ministry will review agencies' budget packages 
and expenditure reports to investigate any 
significant variances that seem unjustified, and 
make adjustments where appropriate. 
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The Ministry will remind all staff of the 
obligation to provide clear documentation that 
supports approvals for changes in funding. As 
well, the Ministry will modify the reconciliation 
process to require confirmation of sufficient 
appropriate documentation for changes in 
funding. 



Ministry Oversight of Custody/Detention 
Facilities 

Regional offices inspect all custody/detention facili- 
ties annually to ensure they comply with the Child 
and Family Services Act (Act) and the Ministry's 
Youth Justice Manual prior to renewing their 
licence to operate. The Act sets out the minimum 
acceptable standards for the provision of residential 
care to children, and the Ministry's justice manual 
includes additional ministry expectations as well 
as requirements under the federal Youth Criminal 
Justice Act relating specifically to youths in custody/ 
detention facilities. We reviewed the licensing 
process and noted that annual inspections were 
being done. However, we noted the following with 
respect to the annual inspection process: 

• For each facility's annual inspection, Ministry 
inspectors are required to complete approxi- 
mately 200 pages of a checklist. They com- 
plete another 200 pages of the checklist every 
four years as part of a more in-depth review of 
facilities' policies and procedures. Although 
thorough, the checklist is often repetitive 
and has not been written in a way that helps 
inspectors focus on systemic issues to gain an 
overall understanding of how well the facility 
operates. We found that the checklist places 
little emphasis on the quality of the program- 
ming and services being offered to youths to 
reduce recidivism, and places more emphasis 
on health and safety standards. 

• There is no distinction in the severity of 
non-compliance issues identified during the 



inspection process. As a result, it was difficult 
to determine whether the decision to issue or 
review a licence was appropriate. All inspec- 
tion files we reviewed indicated numerous 
items of non-compliance with both the Act 
and the manual, but no facility operator 
had its licence suspended or revoked. One 
licence was issued provisionally in December 
2011 due to violations of the Fire Code. The 
Ministry indicated that it prefers to work with 
facilities to make the necessary improvements. 

• We found inconsistency in the quality of 
inspection reports communicated to facility 
operators. Some reports quoted verbatim the 
sections of the Act or manual that were not 
met, whereas other reports described with 
more precision which aspect of a require- 
ment was not being met and often noted the 
particular staff or resident records where the 
violations were identified. While two regions 
made more specific recommendations for 
corrective action, the other region's recom- 
mendations were too general to be useful; for 
example, it typically recommended just that 
the facility implement policies and ensure that 
all staff were familiar with them. 

• We found no evidence that non-compliance 
items were being addressed on a timely basis. 
One regional office we visited confirmed that 
it doesn't follow up on corrective action until 
the next licence-review process, while the 
other two said they followed up soon after. 
However, we generally found no evidence 
on file to show that the Ministry revisited 
the facilities or obtained verification that 
the agencies running them implemented its 
recommendations. The regions considered the 
non-compliance to have been addressed by 
the agency if the agency provided a response 
or a work plan. For a sample of facilities we 
selected for licensing-file review, one-third 

of facilities had had five or more of the same 
violations in the previous year. 
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• The method of data collection and recording 
makes it nearly impossible for the Ministry 
to analyze province-wide inspection data to 
identify and address trends of non-compliance 
or even to monitor a facility's performance 
from one year to the next. For instance, two 
regions entered results into the system for 
each section of the checklist reviewed and 

for each sample tested, while the third region 
aggregated responses from the samples tested 
and entered one set of results into the system 
for each section of the checklist. 

• People working in custody/detention facilities 
are required to get Canadian Police Informa- 
tion Centre (CPIC) clearance, including 
vulnerable-sector screening, when they are 
first hired. There is no further requirement 
for them to have another CPIC check. We 
noted that almost 40% of employees working 
in agency-operated facilities in two of the 
regions we visited were hired at least five 
years ago, and more than 20% were hired 

at least 10 years ago. It is not uncommon for 
public-sector organizations requiring CPIC 
checks on hiring to require updates every five 
years or so. 



RECOMMENDATION 6 



To ensure that the annual facility inspection and 
licensing process results in a safe and secure 
living environment with effective services and 
programs for youth residents, the Ministry of 
Children and Youth Services should: 

• revise the inspection checklist to eliminate 
duplication and place more emphasis on the 
quality of programming and services being 
offered; 

• work toward obtaining more consistency 
in data collection and recording and in 
reporting inspection findings; 

• where significant compliance issues are 
noted, ensure that appropriate and timely 
follow-up is done; and 



• consider requiring that people working in 
youth custody/detention facilities undergo a 
Canadian Police Information Centre check, 
including vulnerable-sector screening, every 
five years and not only at the time of initial 
hiring. 

MINISTRY RESPONSI 

In 2011, the Ministry completed a review of the 
licensing checklist and reduced it by more than 
60%, from approximately 1,500 lines to about 
500. The revised checklist is to be rolled out in 
fall 2012. In addition to the licensing-review 
process, the Ministry will be using other means 
to review the quality of services. For example, 
as part of its Data Strategy, in 2012/13 the 
Ministry will initiate a review of all the Youth 
Justice Services description schedules, which 
define service expectations and form part of the 
transfer-payment agency's contract. Further- 
more, a framework for program evaluation is 
under development to determine whether the 
various programs adhere to the intended design 
elements that reflect the "What Works" litera- 
ture for evidence-based programming that is 
responsive to the needs of youth. 

As a result of the review of the licensing 
checklist, the Ministry is now requiring full 
use of the automated licensing checklist for its 
Youth Justice licensing activities. In addition, 
the Ministry is building a software application 
that will allow division- wide monitoring and 
tracking of trends and identification of where 
corrective action is warranted. As well, in 
April 2012, the Youth Justice Services Division 
approved a Data Management Framework that 
will result in more consistency in all data activ- 
ities for the Division. 

In September 2012, the Ministry provided 
training to managers and licensing staff to 
ensure a clear understanding of legislative and 
ministry requirements in regard to the licensing 
reviews. These requirements include response to 
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non-compliance and the legislative support for 
the Ministry to provide a period of time for the 
agency to meet all requirements. The software 
application being developed will allow the Min- 
istry to monitor the timeliness and effectiveness 
of follow-up work done. 

The Ministry will review best practices 
and consider requiring more frequent Can- 
adian Police Information Centre checks and 
vulnerable-sector screening for people working 
with youths in custody/detention facilities. 
Currently the Ministry's Code of Conduct for 
Youth Justice Services requires staff to report 
any changes to their original security checks. 



PERFORMANCE MEASUREMENT AND 
REPORTING 

The Ministry has stated that it intends to improve 
outcomes for youths through programs and services 
that are responsive to the risks, needs and strengths 
of youths. Its Youth Justice Services Program has a 
number of important objectives: 

• reducing reoffending; 

• preventing youth crime; 

• increasing community safety; and 

• holding youths accountable and creating 
opportunities for at-risk youths through 
rehabilitative programming. 

However, the Ministry has only one perform- 
ance measure for the program — the number of 
youths who reoffend as a percentage of all youths 
tracked, also referred to as recidivism. The target 
recidivism rate changes every year, and is equal to 
the actual rate achieved in the prior year. Recid- 
ivism rates reported by the Ministry have been 
relatively stable over the past five years for youths 
with community-based sentences and have gotten 
slightly better for youths with custody sentences. 
For the 2010/11 fiscal year, the recidivism rate was 
35% for youths with community-based sentences, 
and 59% for youths with custody sentences. We 



noted that two other Canadian provinces track 
youth recidivism rates, but neither has published 
results for comparison. 

The Ministry defines recidivism as a return to 
provincial youth justice supervision or adult cor- 
rectional supervision, within two years, on a new 
conviction that occurs following the completion of 
a youth community disposition (that is, a proba- 
tion order only) or following the completion of 
a youth custody order of six months or more. In 
simpler terms, only youths who have been found 
guilty through the court process and have served 
a sentence longer than six months are tracked for 
recidivism. 

Although we agree that recidivism is an import- 
ant performance measure, and the definition is 
similar to that used for adults by the Ministry of 
Community Safety and Correctional Services, the 
Ministry does not actually track recidivism for more 
than 80% of the youths who have come into contact 
with the Youth Justice Services program. Groups 
excluded from the calculation for the 2010/11 fiscal 
year are all youths held in detention prior to trial, 
all youths diverted from court through extrajudicial 
sanctions, more than 90% of youths sentenced to 
custody and approximately two-thirds of youths 
sentenced to community supervision. The Ministry 
informed us that it excludes these groups because 
studies indicate the Ministry cannot influence a 
person's behaviour in less than six months. 

We believe that maintaining data on the 
reoffend rates for the 80% of youths not being 
tracked would still prove useful. For example, based 
on the extrajudicial files we sampled in our case-file 
review, we noted that one-quarter of youths serv- 
ing extrajudicial sanctions had been in a diversion 
program previously. Some indication of the effect- 
iveness of the diversion program can be obtained 
by analyzing the number of youths in the program 
who reoffended and received either extrajudicial 
sanctions again or a court-ordered sentence. 

Our other observations regarding performance 
measurement and reporting included the following: 
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• The Ministry does not track the programs or 
services attended by each youth in its care. 
As a result, it cannot correlate youths who 
do not reoffend to the specific programs or 
services they received, which might indicate 
the effectiveness of the programs and services 
over time. This might also provide some indi- 
cation of the effectiveness of the 200 transfer- 
payment agencies that provide such services. 

• The Ministry does not monitor the rate of com- 
pliance with court orders, which may be used 
in assessing whether the program is meeting 
its objective of holding youths accountable. 

At the time of our audit, the Ministry was in the 
process of developing additional outcome-based 
performance measures through an evaluation of 
current data to identify what additional data would 
be needed to measure more than just recidivism. 

In addition, in 2010, the Ministry began a pilot 
program of exit surveys for youths upon their 
release from custody/detention facilities in two of 
the regions we visited. The surveys request feed- 
back from youths on programming, safety, basic 
needs such as food and clothing, case management 
and reintegration planning, and overall experi- 
ence at the facility. In the region we didn't visit, 
the Ministry informed us that it also began a pilot 
program of exit interviews with parents and youths 
upon completion of their probation period to collect 
feedback about their probation service experience. 
In discussing this good initiative with the Ministry, 



we were informed that the Ministry hopes to imple- 
ment both types of exit surveys on a province-wide 
basis next year. 



RECOMMENDATION 7 



To enable it to evaluate and report on the effect- 
iveness of the Youth Justice Services program, 
the Ministry of Children and Youth Services 
should expand the measure for recidivism so 
that it captures most of the youths in the pro- 
gram to better enable it to assess which services, 
programs and delivery agencies seem to be the 
most successful over time. 

| MINISTRY RESPONSE 

The Ministry agrees that not all youths are cap- 
tured in its measure for recidivism. The Ministry 
will work with partners, including the research 
community, to review options for reporting on 
expanded measures for recidivism, since cur- 
rently there is no consistent, national definition 
of youth justice recidivism. 

As part of its Data Strategy, the Ministry has 
developed additional Youth Justice outcome 
measures that are broader than recidivism and 
provide a fuller understanding of the impact 
that programs and services have on all youths. 
Tools to track and report on these outcomes are 
under development. 



Chapter 4 



Follow-up on 2010 
Value-f or-money Audits 



It is our practice to make specific recommenda- 
tions in our value-for-money audit reports and 
ask ministries, agencies of the Crown and organ- 
izations in the broader public sector to provide 
a written response to each recommendation, 
which we include when we publish these audit 
reports in Chapter 3 of our Annual Report. Two 
years after we publish the recommendations and 
related responses, we follow up on the status of 
actions taken by management with respect to our 
recommendations. 

Chapter 4 provides some background on the 
value-for-money audits reported on in Chapter 3 
of our 2010 Annual Report and describes the 
status of action that has been taken to address our 
recommendations since that time as reported by 
management. 

For a number of these audits, hearings were also 
held and reports issued by the Standing Committee 
on Public Accounts (Committee). Where a hearing 
was held, we have included a summary of the Com- 
mittee's recommendations in the applicable section 



of this chapter. Our objective in providing this 
additional reporting is to help ensure that action is 
being taken by audited entities to address the issues 
that the Committee raised during the hearing and in 
any subsequent report to the Legislature. Chapter 6 
describes the Committee's activities more fully. 

We are pleased to be able to report that for 
close to 90% of the recommendations we made in 
2010, progress is being made toward implementing 
our recommendations, with substantial progress 
reported for more than 40% of them. 

Our follow-up work consists primarily of inquir- 
ies and discussions with management and review of 
selected supporting documentation. In a few cases, 
the organization's internal auditors also assisted 
with this work. This is not an audit, and accordingly, 
we cannot provide a high level of assurance that the 
corrective actions described have been implemented 
effectively. The corrective actions taken or planned 
will be more fully examined and reported on in 
future audits and may impact our assessment of 
when future audits should be considered. 
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Casino Gaming 
Regulation 

Follow-up to VFM Section 3.01, 2010 Annual Report 



Background 




Casino gaming in Ontario is primarily overseen 
by two Crown agencies, which have different 
responsibilities and an arm's-length relationship to 
each other: 

• The Alcohol and Gaming Commission of 
Ontario (Commission) is the "regulator," 
with a mandate to regulate, license and 
inspect gaming facilities, and to enforce 
gaming legislation. 

• The Ontario Lottery and Gaming Corporation 
(OLG), as the "operator," builds, manages 
and operates, either directly or with private- 
sector partners, Ontario's 24 casinos and 
slot machine facilities at horse racetracks 
(27 casinos and slot machine facilities in the 
2009/10 fiscal year). 

OLG directly operates 19 gaming venues, includ- 
ing 14 at racetracks that have only slot machines 
and five casinos with both table games and slot 
machines (at the time of our 2010 audit, it directly 
operated 22 gaming venues, including 17 slot 
facilities and five casinos) . It also contracts private- 
sector operators to run day-to-day operations at 
one smaller casino and four large "resort casinos," 
which offer more gaming options, higher wagering 
limits and amenities such as hotels, entertainment, 
and meeting and convention space. 



In the 2011/12 fiscal year, OLG casino gaming 
operations generated more than $3.3 billion in 
revenues ($3.4 billion at the time of our audit in 
2009/10) and incurred $2.6 billion in operating 
costs ($2.5 billion in 2009/10), for a net profit 
of $700 million for the province ($900 million in 
2009/10). More than 85% of all revenues are gen- 
erated by slot machines. 

The general public expects casinos and slot 
facilities to be run fairly and honestly. Casino and 
slot facility customers expect slot machines to 
actually pay out the regulated minimums; those 
who play table games, such as blackjack or craps, 
want assurance that casino employees are honest 
and properly supervised, and that the games are 
run fairly. 

In our 2010 Annual Report, we concluded that 
the Commission had adequate systems, policies 
and procedures in place to meet these expectations. 
The Commission's gaming equipment test lab and 
gaming enforcement procedures were adequate to 
ensure the fair operation of gaming equipment, a 
conclusion confirmed by an independent accredited 
gaming test lab that we hired. Our research into 
other jurisdictions, and advice from external 
experts, also indicated that Ontario's regulatory 
framework for casinos offers one of the stronger 
oversight mechanisms in North America. 

However, we also noted in our 2010 audit 
several areas where the Commission's oversight 
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procedures and gaming transparency could be 
enhanced, as follows: 

• Slot machine patrons are very interested 
in the actual payout ratio and in whether 
these payout percentages vary depending 
on the machine type and denomination (for 
example, a one-dollar or a penny machine). 
Some U.S. jurisdictions such as Nevada pro- 
vide this information, but Ontario did not. 

• We noted that patrons would find it difficult 
to locate information on the maximum 
prize payout on certain slot machines — an 
important disclosure should the machine 
malfunction and erroneously award a multi- 
million-dollar jackpot, as occurred twice in 
the two years prior to our audit. In addition, 
the Commission did not require casinos 

to post the odds of winning a jackpot on 
slot machines. 

• In the 2008/09 fiscal year, commission 
inspectors at three of four gaming facilities 
could not meet their goal of inspecting every 
slot machine once a year. In addition, the 
Commission's gaming audit and compli- 
ance inspectors were behind schedule in 
verifying that gaming facilities complied with 
approval requirements and their own internal 
control manuals. 

• In determining registration eligibility for sup- 
pliers and gaming employees, the Commission 
had no policy for dealing with conflicts of 
interest involving related employees work- 
ing in the same casino. It relied, instead, on 
casino and slot facility operators to deal with 
these situations. 

In a related issue, estimates in the 2009/10 
fiscal year were that Ontario residents spent about 
$400 million annually on foreign-based Internet 
gaming websites. Foreign gaming operators do not 
share their revenues with the province, and the 
Commission has no mandate to regulate Internet 
gaming. To help address this, British Columbia and 
Quebec now offer Internet gaming, and, at the time 
of our 2010 audit, OLG indicated that it planned to 
introduce Internet gaming in 2012. 



We made a number of recommendations for 
improvement and received commitments from the 
Commission that it would take action to address 
our concerns. 



Status of Actions Taken on 
Recommendations 



At the time of our follow-up, the Commission had 
taken action on several of the recommendations 
we made in 2010; however, others are taking more 
time to implement. The Commission is in the pro- 
cess of implementing a new approach to regulating 
gaming that aims to target higher-risk areas and 
to set standards while increasing operational flex- 
ibility for gaming operators by allowing them to 
develop their own control activities and business 
processes. The Commission has completed a ser- 
ies of risk assessments related to casino gaming 
facilities and expects to begin introducing new 
standards and requirements by 2013. As a result of 
the Commission's development of a new regulatory 
approach, which is an ongoing and evolving pro- 
cess, a few of our recommendations have yet to be 
substantially addressed. 

The current status of action taken on each of our 
recommendations is as follows. 

CONTROLS OVER GAMES 

Recommendation 1 

To provide more useful information to slot 
machine patrons and better communicate its role 
in ensuring the integrity of gaming in Ontario, 
the Alcohol and Gaming Commission of Ontario 
(Commission) should: 

• make public the minimum 85% slot machine 
payout percentage, a range of actual payouts, 
and the Commission's role in overseeing this, 
similar to the public disclosures made in Nevada 
and New Jersey; and 
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• review its standards and approval processes for 
new and existing slot machines to ensure that 
the maximum prize payouts and odds of win- 
ning are clearly disclosed or readily obtainable 
on each machine. 

To enhance its already strong controls over elec- 
tronic gaming equipment, the Commission should: 

• assess the reasons for its Electronic Gaming 
Branch not meeting its goal of inspecting all 
slot machines annually and, using a risk-based 
approach, assess the implications of this but 
also the need for an annual 100% inspection 
practice; and 

• regularly audit its inventory controls over 
security seals intended to prevent tampering 
with electronic gaming equipment to ensure 
that proper accounting is in place and that 
unaccounted-for seals are immediately detected 
and investigated. 

In addition, to ensure consideration of key risk fac- 
tors relating to table games, the Commission should 
reassess its approval requirements for surveillance 
plans, including minimum surveillance staff levels at 
gaming facilities. To ensure that gaming operators' 
staff who work in key risk areas, such as table game 
dealers and surveillance staff, have sufficient training, 
the Commission should consider whether it should 
require casino staff to meet predefined standards of 
training and competency. 

Status 

The Commission advised us at the time of our 
follow-up that it was still reviewing current policies 
on casino gaming facilities. It also said that the new 
standards and requirements that casino operators 
must meet would begin being implemented in 
2013, although a date for establishing a standard 
for communicating maximum prize payouts and 
odds of winning to slot players had not been set. 
In the meantime, information on the minimum 
85% expected payout percentage is now currently 
available to the public on the Commission and 
OLG websites. 

The Commission also developed risk-evaluation 
criteria for identifying slot machines that require 



inspection. We were advised that the Commission 
does not have a policy to physically inspect 100% of 
the devices annually. Instead, using the risk-based 
approach, it carries out targeted inspections of 
machines identified as high priority. These include 
new machines, those that have been converted or 
changed in any way that might affect the integ- 
rity of the game, and those being removed from 
service. In addition, the Commission conducts 
risk-based random inspections on installed gaming 
equipment. We were informed that the Commis- 
sion is up to date on its risk-based inspections of 
gaming equipment. 

The Commission has also updated its control 
procedures over security seals to include new pro- 
cedures at every gaming facility. At the end of each 
month, an electronic gaming inspector physically 
counts all unused seals, and a regional manager 
reviews a report that identifies all currently applied 
seals and highlights any suspicious numbers for 
further investigation. In addition, the Commis- 
sion updated the Slot History System database in 
July 2011 to keep a historical record of all seals 
entered into the system so as to be able to reconcile 
them. We visited one slot facility and found that 
new inventory controls over seals were in place as 
required. 

The Commission has initiated a number of pilot 
projects in the development of the standards-based 
approach to gaming regulation, including a regula- 
tory review of surveillance plan requirements. 
While the standards and requirements being 
developed do not specifically address surveillance 
staffing levels or casino staff training and compe- 
tency, the Commission did change existing require- 
ments so that performance reporting of employees 
now includes their adherence to required proced- 
ures. We were also informed that risks related to 
the competency of table game dealers have been 
identified, with further relevant standards and 
requirements to be developed as part of the Com- 
mission's review of policies over gaming facilities 
taking place at the time of our follow-up. 
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GAMING AUDIT AND COMPLIANCE 

Recommendation 2 

Given that Ontario's gaming industry is mature and 
there is a high level of gaming facility compliance with 
its regulatory requirements, the Alcohol and Gaming 
Commission of Ontario (Commission) should develop 
comprehensive control-risk frameworks that would 
allow gaming facilities to be assessed individually for 
risk. Such a framework would allow the Commission 
to cost-effectively focus more of its regulatory over- 
sight on higher-risk facilities and less on lower-risk 
ones andyet still achieve a prudent level of oversight. 
In developing these frameworks, the Commission 
should also assess the reasons for and the potential 
impact of its audit and compliance staff not achiev- 
ing the targeted number of audits and inspections of 
gaming facilities. 

Status 

The Commission has implemented a three-phase 
risk-based approach for conducting audits of 
gaming facilities. 

First, it developed an industry-wide risk assess- 
ment in November 2009 for all gaming sites, 
leading to the creation of a risk profile for each 
individual site. Based on the risk profiles, more fre- 
quent audit cycles were established for higher-risk 
sites. We were informed that the risk profiles are 
also updated on a quarterly basis using the latest 
available information. 

Second, various operations within a specific 
gaming site with a greater risk of non-compliance 
were identified to allow the Commission to focus 
its resources not only on sites determined to be 
higher risk, but more specifically on the internal 
operations, such as surveillance, table games, and 
cashiering, of each site deemed to be higher risk. 

The Commission informed us that in June 2012, 
it implemented the third phase of the updated 
audit approach, which involves determining the 
key controls to be selected for testing. In light of 
the government's 2012 Budget, which announced 
the government's intention to explore the further 
privatization of OLG casino facility operations, the 



Commission advised us it would take new operators 
into account in developing its new risk-based audit 
approach, with implementation planned for 2013. 

In addition, the Commission has been 
developing an integrated audit model in conjunc- 
tion with OLG's internal auditors to enhance 
audit efficiency and effectiveness and to minimize 
overlap of the overall audit function between the 
two agencies. We were informed that, because of 
shifting government and OLG priorities, the time 
frame for completion of this integrated model had 
not been finalized. 

GAMING SUPPLIER AND 
EMPLOYEE REGISTRATION 

Recommendation 3 

To ensure that registration and renewal processes 
meet adequate standards for timely completion and 
consistent quality, the Alcohol and Gaming Commis- 
sion of Ontario should: 

• complete its risk-based assessment for stream- 
lining procedures, and establish benchmarks 
and management tracking reports for registra- 
tion and renewal processing times; and 

• establish a policy defining what could constitute 
potential conflict-of-interest situations involving 
gaming assistants and what situations could 
prove problematic. 

Status 

The Commission implemented a new three-stage 
risk-based process in September 2011 to streamline 
the registration and renewals process for gaming 
suppliers and employees. The first stage establishes 
basic eligibility and determines whether further 
investigation is required using a scorecard to 
assess risk. If an applicant's level of risk is scored 
high, the Investigations and Enforcement Branch 
conducts a more rigorous inquiry, collecting more 
in-depth information through interviews and/or a 
full background review. The final stage involves a 
decision about whether to issue the registration or 
renewal. We were informed that the Commission 
will undertake a comprehensive assessment of the 
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new streamlined process after it has been in place 
for one year. 

The Commission also developed the Casino 
Gaming Performance Measures Dashboard as a 
performance management tracking and bench- 
marking tool. The Dashboard is used to report on 
the overall performance of the Commission against 
key indicators, including registration and renewal 
processing times. Benchmarks for the performance 
indicators have been established, and when targets 
are not met, management investigates the causes 
and makes operational adjustments as required. In 
addition, monthly reports using information from 
the Dashboard — such as the number of licences and 
registrations issued as well as average turnaround 
times for processing — are prepared for the Com- 
mission's Board of Directors and also distributed to 
senior management. 

The Commission has yet to establish any policies 
covering potential conflicts of interest involving 
gaming assistants. We were advised that these 
standards and requirements are being developed 
as part of the Commission's overall move toward a 
standards-based approach to regulation. 

SELF-EXCLUSION PROGRAM 

Recommendation 4 

To ensure that gaming facilities adequately deal 
with patrons who may have a problem with or an 
addiction to gambling and those who participate in 
a self-exclusion program, the Alcohol and Gaming 
Commission of Ontario should develop minimum 
standards, policies, and procedures related to self- 
exclusion for use in Ontario's gaming facilities. 
It should also implement a process of periodic- 
ally reviewing gaming facilities' compliance with 
these requirements. 

Status 

The Commission advised us that it consulted with 
key stakeholders regarding various elements of self- 
exclusion and whether those elements should be 
included in policies and programs approved by the 
Board. We were informed that as a result of the con- 



sultations and a scan of best practices across various 
Canadian and foreign jurisdictions, Responsible 
Gaming Standards are currently being developed 
that will also address self-exclusion programs. For 
example, the standards will require gaming oper- 
ators to offer a voluntary self-exclusion program, 
terminate the OLG accounts of self-excluded per- 
sons, remove them from mailing lists, and withhold 
all incentives and promotions for OLG products and 
services during the period of self-exclusion. The 
Commission expects the new standards to begin 
being implemented in 2013, although a date for a 
standard on self-exclusion programs has not been 
set. In the meantime, we were informed that all sites 
currently operate a self-exclusion program that is 
largely consistent with the standards and require- 
ments that will ultimately be put in place. 

We were advised that along with the develop- 
ment of standards, self-exclusion programs will be 
examined on an ongoing basis using appropriate 
regulatory assurance activities such as audits 
and inspections. 

PERFORMANCE REPORTING 

Recommendation 5 

In order to provide the public, including gaming 
facilities' patrons, with meaningful information on 
its regulatory activities, the Alcohol and Gaming 
Commission of Ontario should research other gam- 
ing jurisdictions' best practices in public reporting, 
and expand the information published in its annual 
report and website to ensure that it provides informa- 
tion of use to gaming patrons and to the public with 
respect to its key regulatory activities and results, as 
well as performance information that demonstrates 
the Ontario gaming industry's competitiveness 
and integrity. 

Status 

Although no research was undertaken by the Com- 
mission on best practices in public reporting in 
other jurisdictions, the Commission has developed 
a Casino Gaming Performance Measures Dashboard 
that it now uses to report on its overall oversight of 
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the industry. Benchmark data from the Dashboard 
is used in the Commission's annual report, which 
was expanded in the 2010/11 fiscal year to include 
information such as the number and results of com- 
pliance inspections, testing of electronic gaming 
equipment and systems, and number and results of 
occurrences investigated by casino enforcement. 
We were informed that other key communication 
vehicles such as the Commission's website will be 
reviewed to enhance the information that is avail- 
able to stakeholders. 

OTHER MATTER 

Internet Gaming 

Recommendation 6 

Although the Alcohol and Gaming Commission of 
Ontario (Commission) does not have a mandate 
to regulate Internet gaming, there are proactive 
measures the Commission could take to protect the 
interests ofOntarians in this area until such time as a 
decision is made as to whether Internet gaming should 
be regulated. Given the estimated nearly $400 million 
that Ontarians gamble eachyear with unregulated 
foreign Internet gaming operators that do not pay fees 
or taxes to Ontario, and the recent decision that the 
Ontario Lottery and Gaming Corporation will offer 
Internet gaming in 2012, the Commission should: 

• conduct research into regulatory, technological, 
and oversight best practices used in other juris- 
dictions over Internet gaming available in their 
respective jurisdictions; 

• develop strategies for possible action that can 
effectively regulate and tax or charge fees on for- 
eign operators doing Internet gaming business 
in Ontario; and 

• consider forming alliances with other provinces 
and the federal government to address Internet 
gaming, as is currently being done by some 
other international jurisdictions. 

Status 

In November 2011, the Commission conducted a 
survey of regulatory bodies around the world to 



assess best practices in regulation, technology and 
oversight and governance as they relate to Internet 
gaming. The review identified jurisdictions where 
knowledge-sharing and collaboration should be 
explored because their established regulations align 
with the Commission's own regulatory direction 
for Internet gaming. The Commission also released 
a request for proposals for consulting services to 
develop a regulatory assurance model for Inter- 
net gaming. The request closed in June 2011. A 
resulting contract was executed in September 2011, 
and work on the model began at that time. As of 
July 2012, the OLG was in the process of selecting a 
provider of Internet gaming as a result of its request 
for proposals completed in February 2012. 

We were advised that since the Commission 
lacks the legal power and mandate to regulate 
foreign operators who conduct Internet gaming in 
Ontario, it had not developed strategies or propos- 
als at the time of our follow-up to regulate and tax 
these operators, or charge them fees. 

We were informed that while the Commission 
has formed no new formal alliances with other 
provinces to address Internet gaming, the Com- 
mission remains in regular contact with other Can- 
adian jurisdictions and regulatory bodies through 
the Regulatory Review Committee and other 
organizations such as the Canadian Gaming Regu- 
lators Association and the North American Gaming 
Regulators Association. The Commission also 
advised us that it is in the process of entering into 
memorandums of understanding with other inter- 
nationally based gaming regulators, including those 
involved in Internet gaming, to share information 
on applicants and on regulatory processes and 
standards. For example, the Commission signed a 
memorandum of understanding in March 2012 with 
the Gambling Commission in the United Kingdom, 
which regulates Internet gaming, to allow sharing 
of information and documentation for eligibility 
assessments of applicants, licensees or registrants, 
and for providing joint inspections, investigations, 
and other compliance and regulatory assurance 
activities with respect to gaming. 
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Ministry of Health and Long-Term Care 

Discharge of Hospital 
Patients 

Follow-up to VFM Section 3.02, 2010 Annual Report 



Background 




Ontario hospitals discharged more than 1 million 
patients in each of the last five years, and about 
20% of these patients required post-discharge care, 
either at home (for example, nursing and personal- 
care services such as bathing) or in a long-term- 
care home. 

Hospital physicians determine the medically 
appropriate time to discharge a patient. This 
determination is important because longer-than- 
necessary hospital stays carry risks for patient 
health, including the risk of getting a hospital- 
acquired infection such as C. difficile. As well, 
hospital care costs more than post-discharge care. 
Further, patients staying in hospital longer than 
necessary block access to hospital beds by others 
requiring hospital care. 

As a result, the Ministry of Health and Long- 
Term Care (Ministry), hospitals and Community 
Care Access Centres (CCACs) have introduced 
initiatives to facilitate the discharge of patients 
from hospital. The CCACs also assess eligibility for, 
and arrange access to, both home care and long- 
term-care homes. 

In 2010, we conducted work at three hospitals 
to assess whether they had implemented effective 
processes for the safe and timely discharge of 



patients. The three hospitals we visited were the 
Credit Valley Hospital in Mississauga, now the 
Credit Valley site of the Credit Valley Hospital and 
Trillium Health Centre; St. Michael's Hospital in 
Toronto; and St. Thomas-Elgin General Hospital in 
St. Thomas. 

We concluded that the hospitals were manag- 
ing their discharge processes well in some areas 
and were in the process of changing other systems 
to improve patient flow. However, we noted that 
processes at all three hospitals could be improved 
in some areas. For example, we reported at the time 
that, province-wide, more than 50,000 patients 
who were ready to be discharged waited in hospital 
beds because of delays in arranging post-discharge 
care (also known as waiting for an alternate level 
of care, or ALC) . The total number of days that ALC 
patients were hospitalized increased by 75% over 
the previous five years and at the time represented 
16% of total patient-days in Ontario hospitals. How- 
ever, no one, including the Local Health Integration 
Networks (LHINS), the CCACs or the hospitals, 
was specifically accountable for ensuring that 
community-based services, including home care 
and long-term care, were available when patients 
were ready to be discharged. 

Among our other significant observations: 
• Although quick multidisciplinary team 
meetings on discharge planning activities 
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were held at all three hospitals, physicians 
attended these meetings at only one hospi- 
tal, and CCAC representatives attended most 
meetings at only one other hospital. 
A ministry expert panel recommended that 
hospital physicians prepare a discharge sum- 
mary to communicate patient information, 
such as follow-up appointments, pending 
test results and medication changes, to sub- 
sequent health-care providers. Although dis- 
charge summaries were generally prepared, 
one hospital was consistently very late with 
them. At all three hospitals, a recommended 
reconciliation of medications on admis- 
sion and those on discharge was often not 
prepared, increasing the risk of subsequent 
medication errors. 

At the hospitals we visited, less than 10% of 
total discharges to long-term-care, complex 
continuing care, and rehabilitation facili- 
ties occurred on weekends even though the 
patient could have been discharged then, 
because many of these facilities would not 
accept the patient until the next business day. 
Wait times in hospital for ALC patients 
varied significantly across the province. For 
example, 90% of discharged ALC patients 
in the North West LHIN were placed within 
27 days of being designated ALC, compared 
to 97 days in the North East LHIN. 
There were minimal guidelines on how 
long it should take from hospital referral to 
patient placement in a long-term-care home. 
Of ALC patients waiting province-wide, 90% 
were placed in long-term-care homes within 
128 days, with 50% placed within 30 days. 
At the one CCAC that tracked this data, 
long-term-care homes rejected between 25% 
and 33% of applications because patients 
required too much care or exhibited behav- 
ioural problems. Accepted applicants were 
often just added to a lengthy wait-list. On 
the other hand, patients often did not want 
to go to homes with short or no wait-lists 



because, usually, these facilities were older 

or far from family. 
We made a number of recommendations for 
improvement and received commitments from the 
Ministry and the three hospitals that they would 
take action to address our concerns. 




In the spring and summer of 2012, the Ministry 
and the three hospitals we visited in 2010 provided 
us with information on the current status of the 
recommendations made in our 2010 Annual Report. 
According to this information, significant progress 
has been made in implementing more than half the 
recommendations. While some progress has been 
made on all of the others, the extent of progress 
varies among the hospitals. Fully addressing certain 
recommendations (by, for example, implementing 
a standardized process for moving patients from 
hospital to long-term-care homes) will take more 
time. The current status of the actions taken by the 
Ministry and hospitals is summarized after each 
recommendation. 

PLANNING FOR IN-PATIENT DISCHARGE 

Recommendation 1 

To provide sufficient time for a patient's family and 
other caregivers to prepare for patients' post-discharge 
needs, hospitab should ensure that: 

• key discharge information, such as the patient's 
estimated discharge date and discharge destina- 
tion, is established and documented for every 
patient by the time of admission or shortly 
thereafter, and revised if the patient's condition 
warrants a change in the discharge date; 

• quick round-table discussions regarding 
patients' readiness for discharge are attended by 
key decision-makers from the multidisciplinary 
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team, such as the patient's physician, who is 
responsible for discharging the patient, and if 
the patient is going to a long-term-care home or 
requires home-care services, by a representative 
of the Community Care Access Centre; and 
• the estimated discharge date and discharge 
plans are communicated to patients and their 
families by using visual displays, such as white- 
boards in patient rooms, as recommended by the 
Flo Collaborative. 

Status 

At the time of our follow-up, two of the hospitals 
said key information, such as estimated discharge 
date and destination, is generally established 
for every patient at the time of admission or 
shortly thereafter. One of the two indicated it was 
upgrading its system to make it easier to document 
the expected discharge date after admission. The 
third hospital said it was implementing strategies 
to establish target dates for patient discharge but 
added that such estimates are complex, as patients 
are often required to wait for test results and 
consultations with specialists before the estimated 
discharge date can be determined. 

One of the hospitals said that physicians and 
CCAC representatives generally attend quick round- 
table discussions about patients' readiness for 
discharge. A second indicated that while physicians 
generally attend or can be consulted during such 
discussions, CCAC representatives attend meetings 
only at high-volume in-patient units. The third said 
CCAC representatives attend discharge discussions 
twice a week, but attend fewer of the daily quick 
meetings because of resource constraints. This 
hospital also launched a pilot project in one unit in 
May 2012 that was expected to improve attendance 
by physicians at the daily quick meetings. 

Two of the hospitals indicated at the time of our 
follow-up that in addition to verbal communication 
by staff, they were using whiteboards in patient 
rooms to advise patients and their families of the 
estimated date and plans for discharge. The third 
said that while physicians verbally communicated 



the estimated discharge date to their patients, and 
some units used whiteboards, it expected to have 
hospital-wide communication of discharge dates, 
using whiteboards, in place by fall 2012. 

ARRANGING POST-DISCHARGE CARE 

Arranging for Home-care Services 
and Equipment 

Recommendation 2 

To better ensure that any required home-care services 
are available when eligible patients are ready to 
be discharged, hospitals, in conjunction with their 
Community Care Access Centres (CCACs) and Local 
Health Integration Networks (LHINs), should develop 
timeframes that are standardized within each LHIN 
that provide adequate advance notice of the date such 
services will be needed and keep the CCAC apprised of 
any changes to the required commencement of home- 
care services. 

Status 

All three hospitals noted that they had started to 
take steps to address this recommendation. While 
all three hospitals indicated that they had not yet 
implemented standardized time frames within their 
LHINs to provide advance notice to their CCACs of 
the date patients would need home-care services, 
two said their LHINs were developing a consistent 
discharge process for hospitals within each LHIN. 

One hospital indicated that it was using a deci- 
sion guide developed in conjunction with its CCAC 
and in use at the time of our 2010 audit. The guide 
specifies timelines required to establish home-care 
services and helps ensure that appropriate home- 
care services are in place prior to a patient's dis- 
charge from acute care. This hospital also said that 
it understood that an electronic Resource Matching 
and Referral System had been implemented across 
its LHIN, enabling hospitals to provide advance 
notice directly to the CCAC of the date patient 
services will be needed, as well as any subsequent 
changes in the date or services needed. The hospital 
further noted that it has been working to increase 
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its community referrals through the Community 
Navigation and Access Program, which uses a cen- 
tralized telephone number to connect patients to 
community support services. 

Another hospital noted that implementation of 
this recommendation depended on other organiza- 
tions, and that it was working with its CCAC to 
create new processes, standards and expectations 
around timely discharge of patients requiring 
home-care services. The hospital expected that 
standardized time frames for patient referrals to the 
CCAC would be implemented over the next year. 

The third hospital noted that its CCAC was usu- 
ally involved with patients soon after admission and 
therefore was generally aware of the anticipated 
date of discharge and whether or not home-care 
services would be required. In May 2012, this hospi- 
tal also implemented a system to electronically refer 
patients to its CCAC for assessment. The hospital 
noted that this system will enable it to better track 
how much advance notice it is providing the CCAC 
about patient discharge dates and the home-care 
services they require. 

Arranging for Long-term Care 

Recommendation 3 

To improve the process for admitting hospitalized 
patients to a long-term-care home, the Ministry, 
working in conjunction with the Local Health Inte- 
gration Networks (LHINs), Community Care Access 
Centres (CCACs), long-term-care homes, and hospi- 
tals, should determine the best approach to placing 
a patient in a long-term-care home and establish 
benchmark standards for completing each stage in 
this process, such as determining patient eligibility, 
completing applications to long-term-care homes, and 
the long-term-care homes' processing of patient appli- 
cations. The Ministry should also consider whether 
LHINs should be made accountable for monitoring 
adherence to the target timeframes. 



Status 

At the time of our follow-up, the Ministry said 
it had begun to review information quarterly on 
patients waiting in hospital for a long-term-care 
home and discussing the status with the LHINs. The 
Ministry observed that these reviews, among other 
things, promote the development, dissemination 
and adoption of best practices in hospital discharge 
and long-term-care home placement. As well, the 
Ministry noted that the LHINs are in the process 
of standardizing the referral management process 
province-wide for moving patients from hospitals to 
long-term-care homes. The Ministry expected that 
standardized referral forms would be developed 
by March 2013. Further, as part of the initiative 
to standardize the referral process, one LHIN has 
implemented an electronic Resource Matching and 
Referral System, which enables the referral and 
matching of patients to the earliest available appro- 
priate long-term-care home. Another LHIN has 
implemented a paper-based Resource Matching and 
Referral System. The Ministry expected the other 
LHINs would pilot resource matching and referral 
systems during the 2013/14 fiscal year that were 
expected to shorten the process for placement into 
long-term-care homes. 

The Ministry planned to collaborate with the 
LHINs and the CCACs by fall 2012 to identify the 
steps in the hospital-to-long-term-care home place- 
ment process for which benchmarks or guidelines 
would likely contribute to reductions in the overall 
time to placement. The Ministry also said that 
obtaining data from the Resource Matching and 
Referral System on actual time frames would be 
the first step in creating benchmark guidelines and 
standards for completing each stage in the process 
of placing a hospital patient in a long-term-care 
home. Once target time frames are in place, the 
Ministry expected LHINs to be responsible for 
monitoring the CCACs' performance and reporting 
periodically to the Ministry. 

Two of the hospitals commented that the Home 
First/Wait at Home program has helped reduce 
the number of patients waiting in hospital for 
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placement in a long-term-care home. One said it 
was developing a process to ensure that all options 
for patients are exhausted before they wait in hos- 
pital for a long-term-care bed; the other said that 
it asks the families of people waiting in hospital 
for long-term care to choose five homes within 
five to seven days, and encourages them to select 
homes that have either an available bed or a short 
wait-list. This hospital further noted that although 
the Long-Term Care Homes Act requires homes 
to accept or reject a patient's application within 
five days, it has found that homes usually do not 
respond within that time frame. 

COMMUNICATING INFORMATION TO 
SUBSEQUENT HEALTH-CARE PROVIDERS 

Recommendation 4 

To ensure that medical information essential for the 
continuity and quality of patient care is communi- 
cated in a timely manner to subsequent health-care 
providers, hospitals should: 

• require discharge summaries to be completed 
for all patients in accordance with the Guide 
to Better Physician Documentation developed 
by the Ministry's Physician Documentation 
Expert Panel; 

• establish a target timeframe, such as a max- 
imum of 10 days, for completing discharge 
summaries and forwarding them to the patient's 
family physician or other subsequent health- 
care providers; and 

• consider the use of a medication reconcilia- 
tion template to be completed for each patient 
detailing any changes between the medications 
the patient was taking on admission and the 
medications that the patient will be taking 
post-discharge. 

Status 

The three hospitals said they were completing dis- 
charge summaries for all patients. One noted that it 
had developed a standardized discharge instruction 
sheet based on best practices and feedback from 



family physicians in the community and began 
using it in spring 2012. 

Another hospital said at the time of our follow- 
up that 48% of its discharge summaries were made 
available to a patient's family physician within 
48 hours of the patient's discharge. This hospital's 
target for the 2012/13 fiscal year was to have 
75% of all discharge summaries available within 
48 hours. The other two hospitals required com- 
pleted discharge summaries within 14 days, but one 
was working toward having them available within a 
48-hour deadline. 

With respect to a medication reconciliation 
template, one hospital indicated that its physicians 
currently complete an electronic discharge sum- 
mary that includes information and instructions on 
a patient's previous and new medications. Another 
hospital expected to have a paper-based medica- 
tion reconciliation system fully implemented by 
July 2012. This hospital also planned to implement 
an electronic medication reconciliation system, but 
said it was too early to determine the likely timing. 
The third hospital noted that physicians now com- 
plete a medication reconciliation template and that 
the hospital faxes each patient's new medication 
regimen to his or her family physician and pharmacy. 

The Ministry indicated at the time of our 
follow-up that about 1,000 physicians had direct 
electronic access to hospital discharge summaries. 
Furthermore, eHealth Ontario, in conjunction with 
the LHINs, was introducing systems province-wide 
to enable physicians who use provincially certified 
electronic medical records to directly access hospi- 
tal discharge summaries by 2015. 

HOSPITAL BED AVAILABILITY 

Recommendation 5 

To help reduce the time admitted hospital patients 

wait for a bed: 

• hospitals should review the times and days of 
the week patients are admitted and discharged, 
and arrange patient discharges to allow suffi- 
cient time for beds to be prepared in advance for 
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new admissions, especially for patients arriving 
at known peak admission times; and 
• larger hospitals should assess the costs and 
benefits of implementing a bed management 
system that provides "live" information on the 
status of hospital beds, including which beds are 
occupied, awaiting cleaning, and available for 
the next patient, as well as the reasons for delays 
in placing admitted patients in available beds. 

Status 

At the time of our original audit, one hospital had 
adjusted the timing of patient discharges to better 
match admissions, and reviewed the days of the 
week and times that patients are admitted and 
discharged. It said it was still discharging about 
a quarter of its patients before 11 a.m. However, 
this hospital also noted that various factors affect 
bed availability, including increases in the number 
of emergency-department patients and higher 
demand for specialized services, such as psychi- 
atric and respiratory isolation beds. At the time 
of our follow-up, another hospital said it had also 
analyzed admission and discharge data to better 
understand patterns of patient activity, and had 
aligned services to meet projected bed demand 
where possible. The third hospital said it monitors 
patient discharges before 11 a.m. and before 2 p.m., 
and has introduced a target of having an emergency 
patient in a hospital bed within 90 minutes of the 
decision to admit him or her. 

At the time of our follow-up, all three hospitals 
indicated that they had implemented initiatives 
to help reduce the time that admitted hospital 
patients wait for a bed. In this regard, one of the 
hospitals said its bed management system provides 
"live" information on the status of hospital beds at 
all times, including whether empty beds are await- 
ing cleaning or are available for the next patient. 
Another hospital indicated that its initiatives 
included a system that tracks the status of beds 
from the time one patient leaves a bed until a new 
one occupies it. The third hospital noted that its 
electronic bed management system provides some 



information on the real-time status of beds, but that 
it must use a separate system to let housekeeping 
know about beds that need to be cleaned. In this 
regard, the hospital noted that it had created an 
action group to focus on reducing the amount of 
time that beds are vacant. 

PATIENTS WAITING IN HOSPITAL FOR 
POST-DISCHARGE CARE 

Recommendation 6 

To ensure that patients receive the care they need in 
the location best for the patient: 

• hospitals, in conjunction with their Local Health 
Integration Networks (LHINs), should educate 
all patients and their families on the fact that, 
for patients whose condition has stabilized 

and who no longer need acute care (especially 
older patients), hospitals are not a safe or 
appropriate place to wait for post-discharge care 
(for example, because of the risk of getting a 
hospital-acquired infection such as C. difficile}; 

• the Ministry, in conjunction with the LHINs, 
should assess the costs and benefits of increasing 
the level of post-discharge services that can com- 
mence on weekends to better enable hospitals to 
safely discharge patients on weekends; and 

• the Ministry, in conjunction with the LHINs, 
hospitals, and Community Care Access Centres, 
should give increased consideration to options 
such as more appropriate places for patients to 
safely wait for placement in an alternate-care 
facility such as a long-term-care home; or 
increased supportive-housing arrangements 

to enable patients to continue to live more 
independently. 
Further, to help hospitals better manage their 
patients who are waiting for post-discharge care, the 
Ministry should: 

• further clarify how alternate-level-of-care (ALC) 
wait times should be measured so that ALC wait 
times are being consistently reported to the Min- 
istry's Wait Time Strategy; and 



Discharge of Hospital Patients 




• publicly report the time ALC patients wait 
in hospital before being discharged into a 
community-based setting. 

Status 

At the time of our follow-up, two of the hospitals 
said they had processes to educate patients and 
their families about the risks of a patient waiting 
in hospital for other care when the patient's condi- 
tion has stabilized and he or she no longer requires 
acute care. The third hospital indicated that while it 
does not have a standardized process, some hospi- 
tal staff do discuss this issue with patients. 

The Ministry said it commissioned a compre- 
hensive study in May 2011 of the costs and benefits 
of increasing post-discharge services on weekends 
to facilitate a higher percentage of weekend dis- 
charges from hospitals. The Ministry expects the 
study to be completed by December 2012 and plans 
to share it with the LHINs, hospitals and other 
stakeholders to determine how best to proceed. 
One hospital observed that weekend discharges 
continue to be a problem because most long-term- 
care homes and rehabilitation facilities do not 
accept weekend admissions, which delays patients' 
discharge from hospital. 

The Ministry noted that while it does not have 
information on the number of new spaces created 
for supported housing and assisted living since 
March 31, 2010, it has supported the creation or 
expansion of 28 supported-housing and assisted- 
living services since that time, serving 20% more 
people than before. The Ministry said it has also 
implemented initiatives, including new geriatric 
services and enhanced services for the frail elderly, 
to help seniors live longer in their own homes. The 
Ministry also funded an expansion of the Home 
First program, which provides services that help 
people wait at home until a bed in a long-term- 
care home becomes available. All three hospitals 
supported the Home First program and one com- 
mented that while legislation permits patients 
who must wait in hospital to choose up to five 
long-term-care homes, it is permissible for them to 
make only one choice even if the wait time for the 



home they want is quite long. Two of the hospitals, 
therefore, encouraged patients waiting in hospital 
to choose five long-term-care homes, and one of 
these strongly encouraged these patients to choose 
at least two homes that either have beds available 
or have a short wait-list. 

The Ministry also indicated that hospitals 
are provided with ongoing clarification on how 
alternate-level-of-care (ALC) wait times should 
be measured so that these times are reported in a 
consistent way to the Ministry's Wait Time Strategy. 
At the time of our follow-up, 114 hospitals were 
submitting ALC data. In addition, the Ministry has 
made this ALC information available in real time to 
hospitals, LHINs and CCACs since September 2011. 

Given the other initiatives to address ALC wait 
times, the Ministry noted that public reporting 
of the time ALC patients wait in hospital was not 
considered necessary at the time of our follow-up. 
However, the Ministry said it would monitor the 
need for public reporting on an ongoing basis. 

PERFORMANCE MEASUREMENT 

Recommendation 7 

To help evaluate the patient discharge process, hospi- 
tals should: 

• in conjunction with their Local Health Integra- 
tion Networks (LHINs) and Community Care 
Access Centres, develop measures for monitoring 
and reporting on the effectiveness and safety of 
hospital processes for discharging patients, and 
compare results among hospitals to help identify 
areas for improvement or best practices that can 
be shared with other hospitals; and 

• regularly report key discharge performance 
indicators to senior management and the board 
of directors. 

As well, to help monitor, on a province-wide and 
regional basis, unplanned returns to hospital for the 
same or related conditions, the Ministry, in conjunc- 
tion with the LHINs, hospitals, and the Canadian 
Institute for Health Information, should track post- 
discharge emergency-department visits as well as 
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readmissions to any hospital that occur within a few 
days (or otherwise established reasonable timeframe) 
after a patient is discharged from a hospital. 

Status 

At the time of our follow-up, all three hospitals 
indicated that they were monitoring and reporting 
on certain hospital discharge processes. One said 
it compares its readmission rates for patients with 
certain diagnoses to those of other hospitals across 
Canada. Another said it compares key performance 
measures, including readmission rates, to peer 
hospitals in its LHIN. The third said that while it 
does not perform regular comparisons to other hos- 
pitals, its LHIN compares its percentage of patients 
discharged by 11 a.m. and 2 p.m. each day to the 
percentages discharged from other hospitals as part 
of its Pay-for-Results program. 

All three hospitals indicated that they were 
reporting key discharge performance indicators to 
senior management and their boards of directors; 



at the time of our 2010 audit, only one hospital was 
doing this. 

The Ministry noted that it does not regularly 
track emergency-department visits by recently 
discharged patients for the same or a related 
condition, although it periodically reviews cases 
of people who for any reason visit an emergency 
department within seven days of discharge 
from the same hospital. The Ministry also tracks 
unplanned readmissions to the same or different 
hospitals within 30 days of discharge for all per- 
sons in defined age groups with certain medical 
conditions (for example, people 45 or older with 
congestive heart failure) . The Ministry noted that 
most unplanned readmissions occur through the 
emergency department of the same hospital from 
which the person was discharged. The Ministry 
also indicated that it meets with the LHINs quar- 
terly to review hospital performance, including 
readmission rates. 
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Family Responsibility 
Office 

Follow-up to VFM Section 3.03, 2010 Annual Report 



Background 




All court orders for child and spousal support 
related to divorce or separation proceedings are 
automatically filed with the Family Responsibility 
Office (Office), whose job it is to enforce family- 
support obligations — aggressively if necessary — 
and remit support payments to their intended 
recipients on a timely basis. 

The Office's clients are among society's most 
vulnerable; many of those who have their support 
orders enforced by the Office also collect social 
assistance, often because their former partners 
failed to pay spousal or child support. 

Enforcing court orders for spousal and child sup- 
port can be difficult, and while many willingly meet 
their support obligations, many others go to great 
lengths to avoid making their required payments. 
While acknowledging this, our 2010 audit found 
that the Office was still not successful in achieving 
its mandate of collecting unpaid child and spousal 
support payments. We had a similar conclusion the 
last time we audited this program. We concluded 
in our 2010 Annual Report that the Office must take 
more aggressive enforcement action, enhance its 
case-management process, and improve its informa- 
tion technology and communications systems. As 
well, management needed to work to instill a culture 



of achievement to make the needed changes. Some 
of our other observations at that time included: 

• The Office was slow in following up, where 
necessary, and in registering completed court 
orders for family support. Such delays make 
cases in arrears much more difficult to enforce 
and can result in undue hardship on recipients 
awaiting support payments. 

• Although the Office assigned responsibility 
for each case to an individual enforcement 
services officer, this case-ownership model 
continued to have significant shortcomings, 
including that payers and recipients did not 
have direct access to their assigned officer. 

• Call volumes at the Office's toll-free call centre 
were so high that nearly 80% of calls never got 
through. Of those that did, one in seven call- 
ers hung up before the call was answered. 

• The status of almost one-third of outstanding 
bring-forward notes — intended to trigger 
specific action on a case within one month — 
was "open," indicating either that the notes 
had been read but not acted upon, or that they 
had not been read at all. 

• For ongoing cases, the Office took almost 
four months from the time the case went into 
arrears before taking its first enforcement 
action. For newly registered cases that went 
straight into arrears, the delay was seven 
months from the issue of the court order. 
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• The Office acted in only one in four or one 
in five cases each year to, for example, take 
enforcement action, update case information, 
or track down delinquent payers. 

• The Office had no quality control process 
or effective managerial oversight to assess 
whether enforcement staff have made reason- 
able efforts to collect outstanding amounts. 

• The Office could not provide us with a 
detailed listing by individual account that 
added up to $1.6 billion, which was the figure 
provided to us as the total outstanding arrears 
as of December 31, 2009. 

• The statistical information supplied monthly 
to the Ministry of Community and Social 
Services did not provide a useful summary of 
the Office's successes and failures in collecting 
outstanding support payments or in achieving 
its other key operational objectives. 

• Security weaknesses in the Office's informa- 
tion technology system put sensitive personal 
client information at risk of unauthorized 
access. 

• On a positive note, accounting controls cover- 
ing payments from support payers and the 
subsequent disbursement to intended recipi- 
ents were generally satisfactory, and most 
support payments received were disbursed to 
clients within 48 hours of receipt. 

We made a number of recommendations for 
improvement and received commitments from 
the Office that it would take action to address our 
concerns. 

STANDING COMMITTEE ON PUBLIC 
ACCOUNTS 

The Standing Committee on Public Accounts held a 
hearing on this audit in March 2011. In May 2011, 
the Committee tabled a report in the Legislature 
resulting from this hearing. The report contained 
16 recommendations and requested that the Office 
report back to the Committee with respect to the 
following: 



• the results of its review of a pilot project in 
which a clerk employed by the Ministry of 
the Attorney General had been loaned to the 
Office to process documents passing between 
certain court districts and the Office and so 
reduce backlogs, the impact of the project on 
the Office's enforcement of family-support 
court orders, and whether the project will be 
expanded to other Ontario court districts; 

• recent monthly statistics on calls to the call 
centre (calls answered, abandoned and 
blocked, broken down by local versus 1-800 
calls) and the Office's efforts to report them 
quarterly on its website; 

• recent monthly statistics on office staff absen- 
teeism and attendance trends by branch; 

• the Office's progress in obtaining suggestions 
for improvement from MPP constituency 
office caseworkers; 

• the current status of outstanding bring- 
forward notes (notes requiring follow-up 
action on a case to be taken within a month's 
time), the date by which the Office is to finish 
revising its policies and procedures for the 
proper use of these notes, and the Office's 
plan to routinely check and ensure that staff 
are issuing the notes appropriately; 

• how other jurisdictions approach the enforce- 
ment of support orders to payers whose 
financial circumstances have temporarily and/ 
or unexpectedly declined; 

• any steps the Office will take to determine 
whether enforcement officers are taking the 
most effective and appropriate enforcement 
actions, including using its new management 
system technology to proactively flag recom- 
mended enforcement actions; 

• the Office's progress in negotiating with the 
Ministry of Health and Long-Term Care to 
have OHIP provide it with up-to-date payer 
contact information to help the Office enforce 
support orders; 

• the best option for a caseload management 
model as determined by the Office's review 
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Status of Actions Taken on 
Recommendations 



of caseload management models in other 
jurisdictions; 

• whether the Office is using its capability of 
determining at the end of each month if a 
payer has made his or her support payment, 
and if it is, whether the Office is sending a 
letter informing payers who have not paid 
that they are in arrears and have 15 days to 
respond or face further enforcement action, 
and if it is not, its reasons for not doing so; 

• the results of the Office's analysis of support 
payments in arrears, including the amount 
the Office believes is recoverable and whether 
other jurisdictions can "write off amounts 
deemed unrecoverable; 

• highlights of the Office's expected September 
2011 report on operational performance 
measures and its progress on instilling a more 
results-oriented culture in its workforce; 

• whether the implementation of the Office's 
new case-management system is on schedule 
for spring 2012 and the timeline for the sys- 
tem's key phases; 

• whether the case-management system will be 
able to search other ministries' databases — 
within the bounds of privacy laws — for contact 
information on support payers in arrears; and 

• the Office's strategy for training its workforce 
on each phase of the case-management sys- 
tem and its plans for evaluating whether the 
system meets its business needs. 

The Committee also recommended that the Min- 
ister of Community and Social Services request early 
in the next Parliament that a government motion be 
introduced to establish a Select Committee under 
Standing Order 112(a) to undertake a comprehen- 
sive and comparative review of the Office. 

The Office formally responded to the Committee 
in September 2011. A number of issues raised by the 
Committee were similar to our observations. Where 
the Committee's recommendations are similar to 
ours, this follow-up includes the recent actions 
reported by the Office to address the concerns raised 
by both the Committee and our 2010 audit. 



On the basis of information provided by the Office, 
we concluded that it had made some progress 
on all of our recommendations, with significant 
progress being made on several of them. Signifi- 
cant improvements have been made in registering 
support obligations and in introducing a case- 
management client-service model, and further 
improvements are expected once implementation 
of the new case-management technology system is 
completed. Efforts to reduce payments in arrears 
and suspense account balances and to improve 
performance measurement are ongoing. 

The status of action taken on each of our recom- 
mendations was as follows. 

REGISTRATION OF SUPPORT 
OBLIGATIONS FOR ENFORCEMENT 

Recommendation 1 

To maximize the likelihood of successfully collecting 
support obligations, and to help minimize hardships 
for recipients awaiting their support payments, the 
Family Responsibility Office should: 

• work proactively with family courts in Ontario 
to encourage them to provide complete and 
accurate information on a more timely basis 
so that family-support obligations can be regis- 
tered and enforced more promptly; and 

• register and begin to administer new cases 
requiring no additional information within the 
Office's internal target of 30 days of receipt of 
the court order. 

Status 

In our 2010 Annual Report we noted that the Office 
received court orders, on average, 48 days after 
they had been issued by the courts, and there were 
no ongoing initiatives to encourage the courts to 
forward all support orders or separation agree- 
ments in a timely manner. We also found that on 
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average it took an additional 104 days for com- 
pleted court orders to be registered in the Office's 
information system. 

In its response to our report, the Office indicated 
that it had initiated direct outreach and was provid- 
ing quarterly bulletins to the judiciary in an effort 
to improve the information exchange between the 
courts and the Office, and that it had refined the 
process it used for address verification to enable 
more timely registrations. It also indicated that it 
was initiating two pilot projects, one to provide 
courts with real-time electronic access to its data- 
base to expedite court decision-making on support 
arrears, and one to place a dedicated court clerk in 
its Office to speed up document flow. 

The two pilot projects were initiated in Septem- 
ber and October 2010, respectively. In the first, a 
high-volume court was given access to the Office's 
database in an attempt to expedite judicial decision- 
making by eliminating the need to adjourn cases 
until more financial information was obtained. 
The results were somewhat disappointing, as the 
Office's database was accessed only seven times 
by the court over the course of the 22-month pilot. 
In the second project, a court clerk was accom- 
modated in the Office with ongoing access to the 
Office's database. The Office informed us that this 
project was much more successful, with the clerk 
often being able to process documents in substan- 
tially less time because it was no longer necessary 
to move between the Office and the courts. Many 
documents that typically took months to process 
were done within 24 to 48 hours. Although this pro- 
ject was originally planned to last only five months, 
the Office informed us that it had expanded it to 
include all main issuing courts in Ontario except 
the Superior Court of Justice in Toronto and the 
Ontario Court of Appeal. 

In October 2010 the Office began to distribute a 
quarterly bulletin for the legal community, clients 
and other stakeholders to improve communication 
and information exchange with the courts. The 
bulletins contain details on new Office initiatives 
of possible interest to the courts; tips on drafting 



support orders, dealing with licence suspensions 
and reinstatements, changing domestic contracts, 
enforcing support outside Ontario and other tech- 
nical matters; and success stories about helping 
clients obtain needed support. At the time of our 
follow-up, six bulletins had been issued. 

With respect to registrations, the Office 
informed us that it had conducted a blitz in Janu- 
ary 2012 and successfully eliminated its backlog of 
case registrations. The Office's latest registration 
backlog report indicated that although another 
backlog developed in April and May 2012, it was 
fully addressed in June 2012. As of August 2012, 
the Office's backlog was minimal. 

The Office also engaged a consulting firm in 
March 2011 to review its registration processes 
and help develop new policies and procedures for 
monitoring and following up on registration issues. 
The firm made a number of recommendations, and 
the Office informed us that it had implemented 
several key ones, including streamlining the routing 
of court orders, conducting trace-and-locate efforts 
to ensure the accuracy of client information, and 
calling new clients to let them know what to expect 
from the Office's support programs. 

CASE-MANAGEMENT MODELS 

Recommendation 2 

Given the lack of effectiveness of the current case- 
ownership model in improving the ability of the 
Family Responsibility Office to collect unpaid support 
obligations, the Office should examine processes used 
in other jurisdictions to determine what best practices 
might be applicable to Ontario. 

Status 

In our 2010 Annual Report we concluded that the 
Office's case-ownership model had not been effect- 
ive in improving the collection of unpaid support 
payments, for several reasons. Among these was 
that Ontario payers and recipients did not have 
direct access to their assigned enforcement service 
officer, and that each officer had been assigned on 
average a very formidable 1,377 cases. 
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In its response to our 2010 report, the Office 
indicated that it was moving to a proactive 
case-management model whereby a dedicated 
caseworker would be assigned for the life of each 
client's case and that clients would have easier 
access to their caseworkers and experience fewer 
blocked calls. 

At the time of our follow-up, the Office informed 
us that it had completed an interjurisdictional sur- 
vey of best practices to support payment collection 
in February 2011. Twenty jurisdictions responded 
to the survey, and information gathered from it 
was to be assessed as future business improvements 
were considered. The Office also informed us that 
it had established electronic payment transmission 
linkages with the provinces of British Columbia, 
Newfoundland, Prince Edward Island, Manitoba 
and Nova Scotia, and was working to set up similar 
mechanisms with the remaining provinces. 

The Office also informed us that it had imple- 
mented a new case-management/client-service 
model in November 2011, which was based on its 
2008 pilot project. The new model provides clients 
with direct access to the officer responsible for their 
case. The Office reported that it has been able to 
eliminate its call blockage problem since the model 
was implemented, and call wait times have been 
reduced from an average of eight minutes to an 
average of less than two minutes, while the number 
of calls handled has increased from approximately 
48,000 per month to 80,000 per month. 

CALL-CENTRE OPERATIONS 

Recommendation 3 

Since the call centre remains the primary means by 
which clients communicate with the Family Respon- 
sibility Office, the Office should review its call-centre 
operations and take the steps necessary to ensure that 
all calls are answered within a reasonable time. It 
should also track and report the results of its efforts to 
improve call-centre operations. 



Status 

In our 2010 Annual Report we noted that the Office 
was not regularly monitoring its call centre and 
therefore had little information with respect to the 
number of calls that were not getting through and 
the nature of the calls. A 2008 study found that 80% 
of calls were not getting through the Office's queuing 
system, and one in seven callers who did get through 
eventually hung up before being answered. 

The Office agreed with our recommendations, 
and in its response to our report it indicated that it 
had just implemented a new telephone system in 
June 2010 that provided managers with informa- 
tion to refine the scheduling of call-centre staff and 
to monitor the number of calls not getting through. 
The Office also noted in its response that it had 
developed new customer-service standards that 
would provide benchmarks to measure progress 
and future performance. 

At the time of our follow-up, the Office informed 
us that it had implemented new call-centre 
reporting and monitoring processes in December 
2010, which incorporated performance indicators 
addressing wait-time management, call-taking 
standards, absenteeism and schedule adherence. It 
also informed us that it had implemented customer 
service standards and a feedback process for online 
clients in January 2011. In February 2011, the 
Office used its telephone system to collect informa- 
tion on call blockage rates, the analysis of which 
indicated a correlation between wait times and 
call blockage rates. The Office reported that it was 
able to reduce call blockage rates by 50% after this 
review. However, the implementation of the new 
case-management model in November 2011 moved 
the Office away from a call-centre business model 
and it has since been able to eliminate the problem 
of blocked calls altogether. 

The Office also informed us that it began provid- 
ing a new customer service training program for its 
staff in March 2012. 
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BRING-FORWARD NOTES 



SUPPORT-ENFORCEMENT ACTION 



Recommendation 4 

To help ensure that the Family Responsibility Office 
deals with such issues as client inquiries and enforce- 
ment actions appropriately and on a more timely 
basis, management should monitor whether enforce- 
ment services officers review their bring-forward 
notes, conduct the necessary follow-up work, and clear 
up these notes on a timely and appropriate basis. 

Status 

In our 2010 Annual Report we noted that there were 
approximately 91,000 outstanding bring-forward 
notes awaiting resolution by enforcement officers, 
and that the number of bring-forward notes for a 
sample of these officers ranged from 123 to 1,358 
per officer. Bring-forward notes are intended to 
trigger specific action on a case within one month. 
We also noted that the status of one-third of these 
notes was "open," meaning they either had not 
been read or had not been acted upon, and that 
despite an Office target of addressing bring-forward 
notes within 30 days, about half of the notes had 
been outstanding for more than 90 days. 

In its response to our report, the Office informed 
us it would conduct a blitz in the fall of 2010 to 
address outstanding bring-forward notes, and from 
that point on staff training and new performance 
measures would be developed to ensure that bring- 
forward notes were managed properly and followed 
up in a timely fashion. 

At the time of our follow-up, the Office 
informed us that it had completed its blitz and 
had taken action on all outstanding bring-forward 
notes. It had also introduced a new policy and 
performance measurement mechanism to monitor 
and manage bring-forward notes, and all Office 
staff and managers had been trained on the new 
policy in June 2011. The Office said that it planned 
to eliminate the use of bring-forward notes after its 
new case-management system was implemented, 
replacing it with a process of system-generated and 
manually created service requests to streamline the 
follow-up process. 



Recommendation 5 

To help it collect arrears more effectively, the 
Family Responsibility Office should ensure that 
enforcement staff: 

• initiate enforcement actions for both ongoing 
and newly registered cases on a more timely 
basis; and 

• document why specific enforcement steps were, 
or were not, taken, and concentrate on those 
steps that are apt to be more successful in par- 
ticular circumstances. 

The Office should also establish a quality control 
process and effective managerial oversight to assess 
whether reasonable efforts have been made to collect 
arrears. If it is determined that reasonable efforts 
have not been made, it should take corrective action. 

Locating payers is often the most challenging issue, 
so the Office should also discuss with the Ministry of 
Health and Long-Term Care the current restriction on 
access to payer addresses from the OHIP database. 

Status 

In our 2010 Annual Report we noted that approxi- 
mately two-thirds of all support-payers were either 
in non-compliance or only in partial compliance 
with their support obligations and that enforce- 
ment actions were often neither timely nor effect- 
ive. We found that it took on average almost four 
months after a case went into arrears before officers 
took their first enforcement action, and seven 
months for newly registered cases for which no 
child or spousal payments had ever been made. We 
also noted long gaps between enforcement actions, 
ranging from six months to five years (averaging 
two years) and that only 20% to 25% of the Office's 
total cases were worked on in any given year. 

The Office agreed with our recommendations, 
and in its response to our 2010 report indicated that 
it was updating its policies and procedures to make 
enforcement actions more consistent and effective. 
It also planned to implement new case-manage- 
ment technology in 2012 to enable more proactive 
enforcement actions, and was working with the 
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federal government, law enforcement agencies and 
other provincial ministries to obtain new tools and 
databases for locating defaulting payers. 

At the time of our follow-up, the Office informed 
us that it had, with the assistance of an external con- 
sulting firm, reviewed its processes for default hear- 
ings and warrants of committal in 2009 and then 
implemented changes to both processes to improve 
quality control and oversight. It further informed 
us that in November 2010 it had begun discussions 
with the federal government and law enforcement 
organizations to gain access to new trace-and-locate 
tools, such as the Canadian Police Information Cen- 
tre database. Although this request was eventually 
denied, in December 2010 the Office of the Registrar 
General agreed that the Family Responsibility Office 
could access its database to obtain reports of death 
registrations and name changes for help with locat- 
ing clients and verifying deaths. 

The Office indicated that it had completed a 
report profiling its arrears in February 2011, and in 
April 2011 had initiated a project with the Ministry 
of Health and Long-Term Care (MOHLTC) aimed at 
obtaining access to its Registered Persons Database 
containing the address information of Ontario 
health-card holders. The Office was granted access 
to make unlimited manual address-related quer- 
ies of the database, through a dedicated resource 
at MOHLTC. In May 2011 the Office was granted 
further access to the databases of the Municipal 
Property Assessment Corporation and the Personal 
Property Security Registry. Finally, in August 2011 
the Office's seven trace-and-locate specialists were 
granted unlimited access to the Registered Persons 
Database. Office management informed us that 
this tool has been found to be one of the most use- 
ful for locating payers, and has led to a significant 
improvement in the overall tracing success rate. 

CASELOADS 

Recommendation 6 

To help improve the administration of its enforcement 
program, the Family Responsibility Office should: 



• establish reasonable criteria and benchmarks 
setting out what is a manageable caseload, and 
staff its enforcement activity accordingly; and 

• regularly monitor and assess the productivity 
and effectiveness of its enforcement staff, both 
individually and collectively, in responding 

to inquiries, taking timely and appropriate 
enforcement actions, and collecting outstanding 
support obligations. 

Status 

In our 2010 Annual Report, we noted that enforce- 
ment officers were each handling 1,377 cases on 
average, while enforcement staff in two other large 
provinces operated with average caseloads of 446 
and 312, respectively. We further noted that the 
Office had never established standards for what a 
reasonable case load should be, and had no mon- 
itoring system in place for management to assess 
enforcement staff productivity. 

In its response to our report, the Office informed 
us that efforts were underway to establish a new 
caseload-management model for staff, and that the 
work would be completed by the end of 2010. In 
addition, the new case-management system planned 
to be in place by the end of 2012 was expected to 
equip staff with better tools, such as automated 
reminders, and allow management to better monitor 
the effectiveness of enforcement actions. 

At the time of our follow-up, the Office informed 
us that it had begun tracking productivity meas- 
ures of enforcement staff in February 2011. These 
measures included how quickly calls were being 
answered, the timeliness and appropriateness of 
enforcement actions and the appropriate use of 
bring-forward notes. 

SUPPORT PAYMENTS IN ARREARS 

Recommendation 7 

To enable it to concentrate its efforts on those 
accounts most likely to yield results and to objectively 
measure the effectiveness over time of its enforcement 
activities, the Family Responsibility Office needs to 
obtain better data on support payments in arrears. 
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Status 

In our 2010 Annual Report, we found that the 
total amount of support payments in arrears as of 
December 31, 2009, was approximately $1.6 bil- 
lion — up 23% since our previous audit in 2003. The 
Office had minimal information on this balance, 
and could not provide us with the individual bal- 
ances that accumulated to that total, or which por- 
tion of this balance was deemed uncollectible. 

The Office agreed with our recommendation, and 
in its response to our 2010 report indicated that it 
had developed a number of performance measures 
to strengthen its collection function. These measures 
included the cost of collecting support payments, 
value of arrears owed, number of enforcement 
actions by type, and disbursement rates. 

At the time of our follow-up, the Ministry of 
Community and Social Services informed us that it 
had conducted a cross-jurisdictional scan of arrears 
best practices in November 2010. After this, in 
December 2010, the Ministry launched a project to 
develop the capacity to gather improved data about 
arrears, client behaviour and enforcement effective- 
ness. The project had four deliverables: a profile 
of arrears, an assessment of arrears collectability, 
an assessment of the effectiveness of the Office's 
enforcement actions with respect to collection and 
compliance, and a guideline on how to better target 
cases to generate support payments. The profile of 
arrears was completed in February 2011, and in Sep- 
tember 2011 the project team developed operational 
measures related to arrears, including the cost of 
collecting support payments, the number of enforce- 
ment actions by type, and disbursement rates. 

The Office further informed us that it had begun 
updating policies and procedures related to manag- 
ing and categorizing arrears in March 2012. The 
Office planned to complete this project in late 2012. 

PAYMENT PROCESSING 

Recommendation 8 

While the Family Responsibility Office is generally 
successful in processing and getting most support 



payments to intended recipients on a timely basis, it 
should strengthen its internal controls by: 

• more diligently following up on and clearing 
items in the identified, unidentified, and miscel- 
laneous suspense accounts; and 

• adequately documenting the basis on which 
funds have been released from suspense 
accounts, along with evidence of managerial 
review and approval of the release of such funds. 

The Office should also develop the computerized 
capability to calculate interest on support payments 
in arrears. 

Status 

In our 2010 Annual Report, we noted that 
accounting controls over payments received and 
subsequently disbursed to recipients were gener- 
ally satisfactory. However, we did find that some 
support payments were in "suspense" accounts 
awaiting resolution for various reasons, and that 
these accounts were not adequately controlled. For 
example, we found that the Office failed to follow 
up on or clear almost three-quarters of the items 
in one such account, totalling $2.9 million, within 
the required 90 days of receipt, and that the aver- 
age age of the balances in this account was more 
than nine months. Another suspense account that 
held $2.1 million at the time of our audit contained 
balances with an average age of over three years. 
A third suspense account found not being properly 
addressed contained $7.2 million. We also found 
that the investigations and decisions to release 
funds from these suspense accounts were often not 
adequately documented or approved. 

In its response to our report, the Office indicated 
that it had assigned staff resources to follow up on 
its suspense account balances, and was planning to 
revise its financial policies to incorporate perform- 
ance time frames in an effort to get payments to 
clients more quickly. 

At the time of our follow-up, the Office informed 
us that it had realigned the accounting functions 
in July 2010 and added several financial staff to 
enhance the segregation of duties, increase its 
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analytical capacity and better monitor the client 
support payment process. In October 2010 the 
Office implemented new procedures for sorting 
and analyzing suspense account transactions and 
for documenting the reasons for holding or releas- 
ing funds in these accounts. Further, it developed 
new reports to allow daily monitoring of suspense 
account balances. 

The Office also informed us that it had imple- 
mented a user-log policy for bring-forward notes 
in March 2011, which included expected time 
frames for action on the notes and for payment 
processing. In the fall of 2011 the Office initiated a 
trace-and -locate project for a sample of transactions 
in the miscellaneous suspense account and was 
able to close out some of the accounts and release 
payments to clients. The Office reported that, as of 
March 2012, it had reduced the suspense accounts 
by $2.2 million (44%) and the miscellaneous 
account by $1.4 million (19%) relative to the 
December 2009 balances. 

With respect to interest on support payments 
in arrears, the Office informed us that it does not 
have the legislative authority to calculate interest, 
but that it does pursue interest where a court order 
includes an interest payment provision, when 
claimed by the recipient. 

PERFORMANCE MEASURES 

Recommendation 9 

To help assess whether the Family Responsibility 
Office is meeting its stated objectives, and to help iden- 
tify in a timely manner those areas needing improve- 
ment, the Office needs to define its key operational 
indicators, establish realistic targets, and measure 
and report on its success in meeting such targets. 

Status 

In our 2010 Annual Report, we concluded that 
the Office did not have sufficient information to 
enable it to properly assess its success in meeting 
its operational objectives, or for identifying areas 
in need of improvement. We suggested a number 



of areas where management information would 
be useful, such as the time required to disburse 
funds to intended recipients, the timeliness of vari- 
ous enforcement actions and the length of time 
accounts have been in arrears. 

In its response to our report, the Office indicated 
that it had established a performance measurement 
framework and was working to develop operational 
measures for items such as the cost of collecting 
support payments and the time needed to respond 
to a changed support order. 

At the time of our follow-up, the Office informed 
us that it had completed several performance 
measurement projects in 2010. The first was the 
implementation of an "Executive Dashboard" pro- 
viding management with an overview of key Office 
performance indicators and measures showing 
whether the Office was achieving internal targets 
in areas such as customer service, operations and 
financial strength. In November 2010 the Office 
implemented operational performance measure- 
ment at the branch level to provide managers and 
directors with information to better manage their 
staff and business processes. In December 2010 it 
established a protocol for incorporating its perform- 
ance measures into the new case-management sys- 
tem so that this information would still be available 
when the system went live. 

The Office further informed us that it had 
recruited a senior manager of program effective- 
ness and quality assurance in October 2011 to bring 
greater focus and expertise to the areas of target- 
setting and performance measurement. In January 
2012 the Office finalized a quality assurance frame- 
work incorporating the performance measures that 
had been developed. 

The Office indicated that it would continue its 
work to further define and measure additional pro- 
gram performance indicators in the areas of collect- 
ability, enforcement effectiveness and case profiling. 
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COMPUTER SYSTEMS 

Recommendation 10 

Pending development and implementation of a new 
IT system, the Family Responsibility Office should 
strengthen security requirements and processes for 
its existing IT operations, including the Maintenance 
Enforcement Computerized Assistance system, to help 
better protect sensitive client information. 

Status 

In our 2010 Annual Report, we concluded that the 
Office's main business software was out of date and 
did not adequately support the administration of 
the Office, and a project to develop a new computer 
system had been discontinued after $21 million had 
been spent on it. At the time of our audit the Office 
had been developing another new system, with an 
expected implementation date of April 2012 at a 
projected cost of $49.4 million. 

In response to our report, the Office indicated 
it had taken steps to mitigate risks related to its 



legacy systems, including penetration testing on its 
firewalls and servers and the initiation of a project 
to investigate enterprise file-transfer processes to 
improve controls over information exchanges with 
external organizations. It also affirmed that it was 
on track to deliver a new case-management solu- 
tion in 2012. 

At the time of our follow-up, the Office informed 
us that it had established a quarterly password 
review process to better control and monitor system 
access, and in December 2010 it had its firewalls 
and services located in the Kingston data centre 
tested by OPS Corporate Security. The Office fur- 
ther informed us that it had expanded the use of its 
electronic file transfer service to securely exchange 
information with a number of external organiza- 
tions in March 2011. We were also informed that 
the new case-management solution is close to 
completion and is expected to go live late in 2012. 



Chapter 4 




Ministry of Health and Long-Term Care 



Home Care Services 



Follow-up to VFM Section 3.04, 2010 Annual Report 




Background 



Community Care Access Centres (CCACs) contract 
with service providers to provide home care servi- 
ces to Ontarians who, without these services and 
supports, might need to be admitted to hospitals or 
long-term-care homes. Home care also assists frail, 
elderly people and people with disabilities to live as 
independently as possible in their own homes. 

The CCACs assess potential clients for eligibil- 
ity and approve provision of professional services, 
such as nursing, physiotherapy and social work, as 
well as personal support and homemaking services, 
such as assistance with daily living. CCACs also 
authorize admissions to long-term-care homes. 
In the past three years, home care funding has 
increased 10.5%, from $1.9 billion in the 2009/10 
fiscal year to $2.1 billion in the 2011/12 fiscal year. 
In 2011/12, Ontario provided services to 637,700 
clients, compared to about 600,000 clients in the 
2009/10 fiscal year — a 6% increase. 

There are 14 CCACs in Ontario, each of which 
reports to one of the province's 14 Local Health 
Integration Networks (LHINs). The LHINs, in turn, 
are accountable to the Ministry of Health and Long- 
Term Care (Ministry). 

In 2010, we conducted an audit to assess 
whether mechanisms were in place to meet home 
care needs and ensure that services were provided 
consistently across the province. Our work included 



visits to three of the 14 CCACs (South East CCAC, 
Central CCAC and Hamilton Niagara Haldimand 
Brant CCAC), and we surveyed the other 11 as part 
of our audit. 

We acknowledged in our 2010 audit that the 
Ministry had recognized that enhancing home care 
services offers both cost savings and quality-of- 
life benefits by allowing people to remain in their 
homes. We also noted that home care funding 
had increased substantially since our 2004 audit, 
and independent CCAC client satisfaction surveys 
indicated that home care clients were generally 
satisfied with the services they receive. 

However, we noted that some of the main con- 
cerns identified in our previous audits (in 1998 and 
2004) of the home care program still remained. 
Among our significant findings: 

• Per capita home care funding varied widely 
among the 14 CCACs, resulting in funding 
inequities. Total funding to CCACs had not 
been allocated on the basis of specific client 
needs, or even on a more general basis that 
takes into account such local needs as popula- 
tion size, age and gender of clients, or rural 
locations. 

• Although ministry policy required CCACs to 
administer programs in a consistent man- 
ner to ensure equitable access no matter 
where clients lived, as a result of funding 
constraints, one of the three CCACs we visited 
had prioritized its services so that only those 
individuals assessed as high-risk or above 
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would be eligible for personal support servi- 
ces, such as bathing, changing clothes, and 
assistance with toileting. Clients assessed as 
moderate-risk in this CCAC were deemed not 
eligible, while they would have been eligible 
to receive home care services in the other two 
CCACs we visited. 

• Eleven of the 14 CCACs have a wait list for 
various home care services. The other three 
CCACs said that they had virtually no wait 
lists at all. This is another indicator of a pos- 
sibly inequitable distribution of resources 
among the 14 CCACs. 

• In the absence of standard service guidelines, 
each CCAC developed its own guidelines 

for frequency and duration of services. As a 
result, the recommended time allocated for 
each task and the recommended frequency 
of visits varied, indicating that the level of 
service for people with similar needs may vary 
from one CCAC to another. 

• Although CCACs had made progress in imple- 
menting a standardized initial client-care 
assessment tool, these assessments were often 
not done on a timely basis. 

• Only one of the CCACs we visited conducted 
routine, proactive visits to its service providers 
to monitor the quality of services delivered. 

• CCACs expressed concern with not being able 
to procure services from external service pro- 
viders competitively. The Ministry had asked 
them to suspend the competitive procurement 
process on three occasions since 2002, and, 

at the time of our 2010 audit, the process 
was still suspended. This has contributed to 
significant differences in rates paid to service 
providers for similar services. 

• The 14 CCACs have made good progress 
in implementing an updated case manage- 
ment information system to provide useful 
information to help measure and improve 
performance. 

We made a number of recommendations for 
improvement and received commitments from the 



Ministry and the CCACs that they would take action 
to address our concerns. 



Status of Actions Taken on 
Recommendations 



According to information we received from the 
Ministry and the three CCACs we visited for our 
2010 audit work, some progress has been made in 
addressing all of the recommendations we made 
in our 2010 Annual Report. Most will require more 
time to be substantially implemented. For example, 
CCACs expect to have new Standards of Care to bet- 
ter manage caseloads by March 31, 2013. As well, 
efforts to improve the way funding is allocated to 
CCACs and the way CCACs pay service providers, 
to better reflect client needs and in accordance 
with outcomes, are being phased in. The status of 
the actions taken is summarized following each 
recommendation. 

FUNDING OF HOME CARE SERVICES 

Recommendation 1 

To help ensure that people with similar needs living in 
different areas of the province receive similar levels of 
home care service, the Ministry of Health and Long- 
Term Care, in conjunction with the LHINs, should allo- 
cate funds to CCACs primarily on the basis of assessed 
needs of each local community, using, for instance, the 
proposed Health Based Allocation Model. 

Status 

The Ministry informed us that it is undergoing 
Health System Funding Reform to move away 
from global funding and toward patient-based 
funding. Funding is to be allocated to the CCACs 
using the Health Based Allocation Model (HBAM), 
which estimates expected expenses at the CCAC 
level based on demographic, clinical and financial 
information. Also, funding for some procedures 
would be allocated under another component 
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of patient-based funding called quality-based 
procedures, where health-care providers receive 
funding for the number of patients they treat for 
certain types of procedures on a price-by-volume 
basis, using standard rates for each procedure. 

The Ministry informed us that it has modified 
HBAM to take into account the specific costs of 
each CCAC and to include the components of the 
Resident Assessment Instrument - Home Care tool, 
such as measures of health status, to group CCAC 
clients. It used the revised HBAM in December 2011 
to allocate $27 million in base funding to the LHINs 
to begin addressing some of the historical funding 
inequities among the 14 CCACs. It also applied 
HBAM in the 2012/13 fiscal year to redistribute 
approximately 10% of CCAC base funding while 
ensuring that system stability was maintained and 
access to services preserved. The Ministry also 
informed us that in the 2012/13 fiscal year it would 
use HBAM to determine a portion of the new LHIN 
base-funding allocation for the community sector 
announced by the government in that year. 

The Ministry said that it would continue to work 
with the CCACs to further refine the funding model 
for future years. As part of this process, the Ministry 
plans to increase the portion of CCAC base funding 
allocated under Health System Funding Reform 
until CCACs are receiving 70% of their allocations 
by patient-based funding, by the 2014/15 fiscal year. 
The Ministry also indicated that it would provide 
supports to inform and assist CCACs and LHINs with 
the transition to the new funding approach. 

DELIVERY OF HOME CARE SERVICES 

Case Management Caseloads 

Recommendation 2 

To ensure that case managers are deployed optimally 
and to encourage equitable service levels across the 
province, the Ministry of Health and Long-Term Care 
should work with LHINs and the Ontario Association 
of Community Care Access Centres to establish case 
manager-client caseload guidelines. 



Status 

According to the Ministry and the three CCACs, 
the new Client Care Model under the Integrated 
Client Care Project (ICCP) is redesigning the way 
health-care services are provided. The Client Care 
Model uses a population-based case management 
approach developed by the Ontario Association 
of Community Care Access Centres. Clients are 
categorized into different populations and sub- 
populations based on factors such as health condi- 
tions, degree of independence and risk of acute 
episodes. The model enables case managers to 
specialize in specific client populations, co-ordinate 
client care across the entire health system and 
monitor the impact of the care provided. 

The three CCACs informed us that, also as part 
of ICCP, they are working to implement Standards 
of Care, case manager-client caseload guidelines 
outlining the role and expectations for client servi- 
ces staff by client population and sub-population. 
The three CCACs informed us that all 14 CCACs 
were categorizing clients according to consistent 
definitions of populations and sub-populations to 
enable caseload realignment among case managers. 
The standards in place at the time of this follow-up 
had been approved in July 2011 by all CCACs. They 
are to evolve as future sub-populations are identi- 
fied and as implementation occurs. CCACs antici- 
pated complete implementation of the standards by 
March 31, 2013. 

At the time of our follow-up, the Client Care 
Model was being applied to four areas of care: indi- 
viduals requiring wound care; individuals requiring 
palliative care; frail seniors; and medically fragile 
children. The model focuses on high costs and/ 
or volumes and the potential to improve care 
and either reduce the cost of care or increase the 
amount of care provided for the same cost. 

The Ministry informed us that it was testing a 
new wound care model using outcome-based path- 
ways and outcome-based payments at four CCACs, 
with a goal to apply the new model at all CCACs 
and for all types of wounds. Five palliative care test- 
ing sites have also been launched, and the CCACs 
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are to begin initial research for outcome-based care 
pathways for frail seniors and medically fragile chil- 
dren in April 2013. Participating CCACs and service 
providers are expected to be identified at that time. 

According to the Ministry all work completed as 
part of the ICCP is being assessed by external evalu- 
ators to identify the best policy options for health 
care and health-care funding and incorporate them 
into the system. 

Admission to Services or Wait-lists, 
Service Levels, Monitoring Home Care 
Services Provided, Client Reassessment for 
Continued Services 

Recommendation 3 

To help ensure that an appropriate and consistent 
level of service is provided to home care clients, Com- 
munity Care Access Centres should: 

• monitor case manager adherence to the 
established timelines for both the initial client 
assessment and the periodic client reassessments 
and, where such timelines are not met, ensure 
that case managers document the reasons in the 
applicable client files; 

• enhance external provider oversight to better 
ensure that the expected and paid-for levels of 
service are being provided to home care clients; 
and 

• regularly review both client complaints and 
client events to identify any systemic areas 
requiring further follow-up. 

To promote equitable funding and service levels 
across the province, the Ministry of Health and 
Long-Term Care, in partnership with the LHINs, 
should consider incorporating summary data from 
the standardized Resident Assessment Instrument to 
assist in developing a more client-needs-based funding 
model and to encourage the CCACs to adopt consist- 
ent criteria for prioritizing the differing levels of home 
care services. 

Status 

According to the three CCACs, work was underway 
to enhance the Client Health Related Information 



System (CHRIS) to support compliance with the 
provincially endorsed Standards of Care at the 
individual client level. This should also enable 
better monitoring of client assessments and re- 
assessments. The improved system will remind case 
managers when assessments are due by client care 
model and population type according to the Stan- 
dards of Care. The enhancements to CHRIS began 
to go live in October 2012. 

The three CCACs indicated that they have 
been monitoring reassessment standards through 
various means while they waited for the CHRIS 
enhancements to be fully implemented. The first 
CCAC has been conducting manual audits of the 
files of clients who have been identified as not hav- 
ing had an assessment when planned, to ensure 
that the reasons for this are valid and documented. 
The second CCAC has been tracking the frequency 
at which standards are met at the caseload level 
and the organizational level to help pinpoint prob- 
lem areas. The third CCACs case managers were in 
the process of setting timelines for the frequency 
at which reassessments should occur, according 
to type of client population. It had completed its 
review of caseload files by July 2012 and expected 
to adjust the frequency of its reassessments to 
match the new standards of client care for CCACs 
by the end of November 2012. 

The three CCACs informed us that they have 
adopted an audit framework for the oversight 
of contracted service providers. The framework 
includes a process for risk evaluation to pinpoint 
areas that provincial audit activity should focus on. 
In 2011 an audit was conducted at three CCACs 
on the identification, reporting, management and 
quality-improvement processes related to missed 
visits. Audit processes for assessing compliance and 
quality of services were to be further developed by 
a provincial Missed Visit Working Group through- 
out 2012. 

According to the three CCACs, the Ontario Asso- 
ciation of Community Care Access Centres main- 
tains a province-wide reporting site that captures 
data that can be used to assess service provider 
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performance. All three CCACs indicated that they 
regularly review performance data with their ser- 
vice providers and discuss areas where the quality 
of service could be improved. They also indicated 
that work was underway to further develop the per- 
formance measures to better support the monitor- 
ing of client outcomes; their target was to introduce 
public reporting through Health Quality Ontario at 
the service provider level by the 2013/14 fiscal year. 

All three CCACs informed us that they track 
complaints and events and monitor these for areas 
requiring follow-up. The 14 CCACs developed a 
provincial common events framework to standard- 
ize the way in which client complaints and certain 
key events are tracked and managed across their 
sectors. The three CCACs informed us that 13 of the 
CCACs had finished aligning their data capture with 
the new events framework, and that the remaining 
CCAC would align its data in the future. 

As noted earlier, the Ministry is moving toward 
patient-based funding and has been working with 
the CCACs to enhance the Health Based Allocation 
Model (HBAM), which uses demographic, clinical 
and financial information to estimate expected 
expenses at the CCAC level. The Ministry informed 
us that part of the enhancements to HBAM included 
incorporating relevant components of the Resident 
Assessment Instrument - Home Care to allocate 
funding for long-stay clients. This includes data 
such as measures of health status, which can 
be used to group clients and related costs. Also, 
as noted earlier, CCACs have been working on 
developing and implementing Standards of Care, 
which would establish consistent criteria for priori- 
tizing levels of home care services. 

Acquisition of Services from Contractors 

Recommendation 4 

To ensure that home care services are procured from 
external providers in a cost-effective manner, the Min- 
istry of Health and Long-Term Care should work with 
LHINs and the Ontario Association of Community 
Care Access Centres to: 



• formally evaluate the expected cost savings from 
allowing CCACs to procure home care services on 
a competitive basis, keeping in mind the poten- 
tial impact on clients and service levels; and 

• in the meantime, conduct a review of service- 
provider rates by type of service across Ontario 
to determine whether the significant rate varia- 
tions are warranted in relation to the actual cost 
of providing the service. 

Status 

The Ministry informed us that government direction 
for CCACs to proceed with competitive procurement 
was still pending at the time of our follow-up, thus 
the CCACs had not yet been able to start procuring 
home care services on a competitive basis. The 
Ministry also informed us that the CCACs are work- 
ing with their providers to ensure that the providers 
are aware of performance expectations, are focused 
on continuous quality improvement as part of their 
core business, and have appropriate tools, training 
and information to support quality delivery. 

The Ministry indicated that a key design element 
of its Integrated Client Care Project (ICCP) was the 
development of an alternative payment process. 
Following the new process, payment to contracted 
home care service providers would be based on 
their achievement of specific client outcomes using 
evidence-based, best-practice care pathways. At the 
time of our follow-up, this alternative payment pro- 
cess was being tested at two wound care sites. First, 
wound care milestones are identified (for example, 
a wound must be 30% healed in 30 days), and then 
payment to the provider is to be based on achieving 
these milestones. The CCACs noted that the work on 
the alternative payment process would inform the 
Ministry's patient-based funding initiative as well as 
the contracting approach in the future. According to 
the Ministry, patient-based funding addresses rate 
variations, as payment is based on what it refers to 
as evidence-based best-practice care pathways as 
opposed to units of discrete service. 

The Ministry informed us that ICCP has also 
developed the necessary changes to the electronic 
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information system to accommodate a new type 
of billing. Instead of units of home care service 
(volume of service), billing will be done by service 
bundles tied to client outcomes and established 
payment amounts. The billing changes were to be 
rolled out for wound-care populations at half of the 
CCACs in October 2012, with full implementation 
to take place by April 2013. According to the Min- 
istry, this billing approach will be applied to pallia- 
tive care around April 2013 and to other categories 
of care as the care pathways for them and related 
payment amounts are determined. 

Building on the work conducted under the 
ICCP, the CCACs launched an initiative in April 
2012 referred to as Quality and Value in Home 
Care. Under this initiative, CCACs are working with 
service providers and their respective associations 
to review contract requirements. The reviews are 
to encompass integration of care; outcome-based 
methods of reimbursement; service provider per- 
formance indicators; contract management that is 
based on achieving client outcomes; market share 
allocation; geographic boundaries; and volumes of 
service. 

DATA MANAGEMENT AND ANALYSIS 

Recommendation 5 

To reap the full benefit of the recent improvements 
to the case management information system, the 
Ministry of Health and Long-Term Care, working with 
the LHINs, should review the summary-level data 
on a province-wide and regional basis as a means of 
enhancing its oversight of the home care services cur- 
rently being provided. 

Status 

The Ministry informed us that it has conducted 
annual standard education sessions based on recom- 



mendations from the CCAC sector. The last informa- 
tion session, held in March 2012, was to help the 
Ministry and the CCAC sector to obtain a greater 
understanding of the issues around data accuracy, 
consistency and outliers as well as the processes 
used to address these issues. The Ministry indicated 
that all 14 CCACs participated in the session. 

According to the Ministry, LHIN finance and per- 
formance teams are using the CCAC data for com- 
parative purposes and performance monitoring. For 
example, the financial reviews included utilization 
or cost comparisons by functional centre and com- 
parative information about how and where each 
CCAC spent its funds compared to other CCACs. 

According to the Ministry, LHINs monitor indi- 
vidual CCAC activity on a monthly and quarterly 
basis, using dashboards to compare performance to 
expected results. LHINs also monitor CCAC activ- 
ity against other CCACs to review and compare 
performance. Comparisons are made in areas such 
as case management full-time equivalents; nursing 
volumes, personal-support volumes and home- 
making volumes; therapy volumes and expendi- 
tures; and school services utilization. 

The three CCACs informed us that they report 
individually to their LHINs on indicators required 
under their accountability agreements, as well 
as any additional indicators that LHINs require 
individual CCACs to regularly report on, based on 
the priorities in their geographic area. CCACs also 
indicated they are also often asked to provide addi- 
tional, ad hoc reports to LHINs and to the Ministry. 
Various home care quality indicators for long-stay 
clients — such as community wait times, cognitive 
functions and client satisfaction with home care 
services — are reported through Health Quality 
Ontario's website. 
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Background 




Hospital emergency departments provide medical 
treatment for a broad spectrum of illnesses and 
injuries to patients who arrive either in person or 
by ambulance. The quality and efficient delivery of 
patient care in emergency departments depend on a 
variety of interrelated elements, such as prompt off- 
loading of ambulance patients, quick and accurate 
triage (that is, the process of prioritizing patients 
according to the urgency of their illness or injury), 
nurse and/or physician assessment, diagnostic and 
laboratory services, consultations with specialists, 
and treatment. In the 2011/12 fiscal year, there 
were about 5.9 million emergency-department 
visits (5.4 million in 2009/10) in Ontario, at a 
cost of approximately $1.1 billion ($960 million in 
2009/10). 

The public suspects that the main underlying 
causes of what can be lengthy emergency- 
department wait times are walk-in patients with 
minor ailments and hospital administrative issues 
such as understaffed emergency departments. 
However, our 2010 audit found that lack of avail- 
able in-patient beds for emergency patients requir- 
ing hospitalization may well have had an even 
greater impact on emergency crowding and wait 
times. While the Ministry and the hospitals had 



been actively attempting to address the problem at 
the time of our audit, emergency-department wait 
times had not yet shown significant improvement 
or met provincial targets, especially for patients 
with more serious conditions. In our 2010 Annual 
Report, some of our more significant observations 
were as follows: 

• The Canadian Triage and Acuity Scale (CTAS) 
guidelines recommend that patients be 
triaged within 10 to 15 minutes of arrival at 
the emergency department. Yet at all three 
hospitals we visited, some patients waited 
more than an hour to be triaged. 

• In about half of the triage files reassessed by 
nurse educators, the CTAS levels originally 
assigned by triage nurses were found to be 
incorrect. Of these, the majority were under- 
triaged, underestimating the severity of the 
patients' illnesses or injuries. 

• Provincially, only 10% to 15% of the patients 
with emergent and urgent conditions were 
seen by physicians within the recommended 
timelines, and sometimes these patients 
waited for more than six hours after triage 
before being seen by nurses or physicians. 

• At the three hospitals we visited, the timeli- 
ness of accessing specialist consultations 
and diagnostic services affected emergency- 
patient flow. More than three-quarters of the 
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hospitals that responded to our province-wide 
survey of 40 hospitals indicated that limited 
hours and types of specialists and diagnostic 
services available on-site were key barriers to 
efficient patient flow. 

• At the time of our audit, emergency- 
department patients admitted to in-patient 
units across the province spent on average 
about 10 hours waiting for in-patient beds. 
Some waited 26 hours or more. Delays in 
transferring patients from emergency depart- 
ments frequently occurred because empty 
beds had not been identified or cleaned on a 
timely basis. 

• Two of the three hospitals we visited had dif- 
ficulty finding staff to fill nursing schedules, 
especially for night shifts, and on weekends 
and holidays. A number of emergency- 
department nurses worked significant 
amounts of overtime or took extra shifts, 
leading to additional costs and increasing the 
risk of burnout. 

• Paramedics often had to stay in emergency 
departments for extended periods of time 
to care for patients waiting for emergency- 
department beds or until emergency- 
department nurses could accept them. 

• Province-wide, about half of emergency- 
department visits were made by patients with 
less urgent needs who could have been sup- 
ported by alternatives such as walk-in clinics, 
family doctors, and urgent-care centres. 

We made a number of recommendations for 
improvement and received commitments from the 
Ministry and the three hospitals we visited that 
they would take action to address our concerns. 



Status of Actions Taken on 
Recommendations 



According to information provided to us by the 
Ministry and the hospitals, good progress has been 
made in implementing most of the recommenda- 
tions we made in our 2010 Annual Report. Efforts 
to identify reasons for excessive wait times and 
to reduce them have resulted in some improve- 
ments in patient flow, and these efforts continue. 
New legislative requirements have expanded the 
reporting that hospitals do on the quality of care 
they provide. However, the hospitals indicated 
that ensuring that there are sufficient nurses and 
physicians to adequately staff their emergency 
departments remains a challenge. The Ministry and 
the hospitals indicated that it will therefore take 
additional time to fully address some of our recom- 
mendations. The current status of the actions taken 
on each of our recommendations is as follows. 

ONTARIO'S WAIT TIME STRATEGY FOR 
EMERGENCY DEPARTMENTS 

Recommendation 1 

To ensure that emergency departments are operating 
in the most effective way to provide high-quality 
emergency care as quickly as possible to all patients: 

• hospitals should identify causes of delays in 
patient flow and examine ways of reducing wait 
times in emergency departments accordingly; 

• the Ministry of Health and Long-Term Care 
should work with the LHINs and with hospitals 
to identify and disseminate best practices from 
Ontario and other jurisdictions; and 

• the Ministry should provide funding to hospitals 
in a timely manner to enable hospitals to have 
adequate time to implement the funded initia- 
tives cost-effectively. 

Status 

All three hospitals we visited informed us that 
they have taken actions to identify causes of delays 
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in patient flow and examine ways of reducing 
wait times in emergency departments. One of the 
hospitals has developed strategies that include 
promoting the use of its urgent care centre to the 
community as an alternative to the emergency 
department and reviewing discharge patterns in 
the in-patient units. Another hospital has created 
a Process Improvement Office with a focus on the 
root-cause analysis of delays. The third hospital has 
introduced a Medical Assessment and Consultation 
Unit for rapid assessment, treatment and admission 
of emergency-department patients. 

The Ministry indicated that it has expanded 
and continued the Pay-for-Results program and 
the Emergency Department Process Improvement 
Program (ED PIP) in the 2010/11 and 2011/12 
fiscal years. The ED PIP trains staff on best practices 
and supports hospitals in funding the implementa- 
tion of best practices and local solutions to improve 
patient flow and reduce wait times. 

• As part of the ED PIP, staff teams across 

81 hospitals were trained in the use of Lean 
methodology (a methodology focused on 
process speed, efficiency and elimination of 
waste) and engaged in follow-up activities 
to improve patient flow in their emergency 
departments and in-patient units by eliminat- 
ing duplicative or unnecessary steps. The 
Ministry told us that it has also developed a 
comprehensive toolkit and web resources for 
hospital use that will be maintained by Health 
Quality Ontario (formerly the Health Quality 
Council of Ontario) . 

• The Pay-for-Results program, covering 74 hos- 
pitals, continues to have a positive impact 

on reducing the wait times at emergency 
departments for patients with minor condi- 
tions. Ninety percent of these patients were 
treated within 4.2 hours, almost achieving 
the four-hour target, according to the most 
recent data in June 2012. For patients with 
complex conditions, in June 2012 the longest 
that 90% of them could possibly spend in 
emergency departments was 10.3 hours, an 



improvement of about 26% as compared to 
14 hours in April 2008, but still above the 
eight-hour target. 
In order to ensure that hospitals receive funding 
to implement the initiatives in a timely manner, the 
Ministry has created a Transfer Payment Operating 
and Capital Funding Packages Roadmap to stream- 
line its funding processes and increase operational 
efficiencies. The Roadmap is an interactive refer- 
ence document that shows hospitals how to prepare 
a Transfer Payment Funding Package by providing 
connections to the necessary templates, documents 
and information. 

TRIAGE PROCESS 

Recommendation 2 

To ensure that triaging is done appropriately and 
consistently within the recommended timeframe: 

• hospitab should conduct periodic audits to 
monitor the quality and accuracy of triage and 
identify areas for improvements; 

• hospitab should consider performing a quick 
"pre-triage" on patients who cannot be tri- 
aged immediately upon arrival at emergency 
departments; 

• the Minbtry of Health and Long-Term Care 
should work with the LHINs and with hospitab 
to assess whether the reported length of stay at 
emergency departments should include the time 
that patients wait for triage; and 

• the Minbtry should work with the Emergency 
Medical Services (EMS) to provide updated train- 
ing for paramedics to ensure that hospitab and 
paramedics are using consbtent triage practices. 

Status 

All three hospitals we visited stated that they have 
conducted triage audits on a routine basis and 
received positive results. For example, one hospital 
informed us that its triage audits have consistently 
demonstrated a level of completeness and reliability 
of about 80%, which is a significant improvement 
over the 44% reported in our 2010 audit. In order 
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to reduce the burden of auditing triage records and 
to improve the completeness and reliability of tri- 
age, one of the hospitals implemented an electronic 
triage system in February 2012 that includes 
automated prompts and mandatory fields in triage 
records. All three hospitals we visited in 2010 
have either put "pre-triage" in place or made other 
improvements in their triage process. For example, 
one hospital has introduced a process called "walk 
the line," which requires the triage nurse to walk 
through the waiting area to take patients at the 
greatest risk to triage first, and another hospital has 
renovated the design of its emergency department 
to ensure that staff can see all patients in the wait- 
ing area. 

Regarding the time from arrival to triage, the 
Ministry has obtained advice from Cancer Care 
Ontario (CCO), which has managed emergency- 
department data through the National Ambulatory 
Care Reporting System (NACRS). CCO identified 
technical limitations in collecting accurate patient 
arrival times at emergency departments. In 
August 2011, CCO conducted an analysis and found 
that monitoring and reducing the time from patient 
arrival to triage would have a negligible effect on 
the total length of stay in emergency departments. 
Therefore, the Ministry will continue to focus on 
capturing the wait time after triage or registration, 
whichever is performed first. 

To ensure that hospitals and paramedics 
use consistent triage practices, the Ministry has 
developed a Pre-hospital Canadian Triage and 
Acuity Scale (CTAS) Paramedic Guide, which will 
support paramedics in assigning, communicating 
and documenting the appropriate CTAS levels. At 
the time of our follow-up, the Ministry expected 
that the guide would be released at the end of 
August 2012 and would be posted on the Ministry's 
website and distributed to all municipalities 
(municipalities have the primary responsibility for 
dispatching land ambulances). 



ASSESSMENT AND TREATMENT 

Recommendation 3 

To ensure that patients receive timely assessment 
and treatment and an appropriate level of care at 
emergency departments: 

• hospitals should work with the respective LHINs 
to develop, document, and implement proced- 
ures for monitoring and reassessing the status 
of patients in the time interval between triage 
and treatment in accordance with their assigned 
triage level; and 

• the Ministry of Health and Long-Term Care 
should encourage hospitals to track critical 
quality-of-care measures with respect to the most 
serious time-sensitive illnesses commonly seen in 
emergency departments and consider the applic- 
ability of protocols or best-practice guidelines for 
those illnesses on a system-wide basis. 

Status 

Regarding the procedures for monitoring the status 
of patients in the time interval between triage and 
treatment, only one of the hospitals we visited 
indicated that it has a reassessment triage nurse in 
place to reassess patients waiting to be seen on a 
regular basis and to alert physicians immediately if 
a patient's status changes. The other two hospitals 
have relied on staff to follow their existing guide- 
lines in reassessing the condition of patients. 

The Ministry indicated that the following actions 
have been taken subsequent to our audit to ensure 
that patients receive a timely, high-quality and 
appropriate level of care at emergency departments: 

• Through the Excellent Care for All Act, 2010, 
the Ministry has required every public 
hospital to establish a Quality of Care Com- 
mittee that reports on issues related to the 
hospital's quality of care. All public hospitals 
are required to submit their annual quality 
improvement plans to Health Quality Ontario 
and to make the plans available to the public. 

• As part of the Pay-for-Results program, the 
Ministry has continued to require all hospitals 
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with 30,000 or more annual emergency- 
department visits to submit information on 
quality-of-care indicators regularly to their 
Quality of Care Committees. 
• The Ministry has held regular meetings with 
the CEO at each LHIN to discuss wait times and 
quality of care at the hospitals in each LHIN. 

CO-ORDINATION WITH OTHER HOSPITAL 
DEPARTMENTS 

Recommendation 4 

To better allow hospitals to assess the impact that 
timely specialist consultation and diagnostic services 
have on patient care, especially for high-acuity 
patients, hospitals should track targeted and actual 
wait times for specialist consultation and diagnostic 
services for emergency patients, so that the impact of 
these wait times on providing timely and appropriate 
patient care can be periodically assessed. 

Status 

All three hospitals we visited informed us that they 
have tracked wait times for specialist consultation 
and diagnostic services for emergency patients. 
They have also taken actions to reduce wait times. 
One of the hospitals has increased its ultrasound 
hours to serve the needs of emergency patients, 
completed a study to identify areas for improve- 
ment and developed a work plan to improve those 
areas accordingly. Another hospital's emergency 
department has worked with its diagnostic imaging 
department to identify waste and redundancies in 
the process of obtaining diagnostic services. The 
third hospital has implemented a Model for Special- 
ist Consultation to ensure that high-acuity patients 
from its emergency department have prompt 
diagnosis and treatment plans. 

According to the Ministry, through the Emer- 
gency Room/National Ambulatory Care Reporting 
System (ER/NACRS) Initiative led by Cancer Care 
Ontario (CCO), emergency departments have been 
collecting data relating to specialist consultation 
since April 2011. Monthly reports have been 



provided to the Ministry, hospitals and LHINs to 
assist hospitals in assessing the impact of specialist 
consultations on the total time patients spent in the 
emergency departments. As well, the Ministry and 
CCO are currently evaluating the feasibility of and 
technical requirements for collecting wait-time data 
for MRI and CT scans for emergency patients. The 
Ministry informed us that preliminary data would 
be available by the summer of 2014. 

PATIENT DEPARTURE FROM THE 
EMERGENCY DEPARTMENT 

Recommendation 5 

To ensure that vacant in-patient beds are identified, 
cleaned, and made available on a timely basis to 
admitted patients waiting in emergency departments: 

• hospitals should have an effective process in 
place to identify vacant beds and communicate 
their availability between in-patient units and 
emergency departments; and 

• the Ministry of Health and Long-Term 
Care should work with the LHINs and with 
hospitab to identify and disseminate best 
practices that enable hospitals to reduce 
unnecessarily long stays of admitted patients in 
emergency departments. 

Status 

All three hospitals we visited have put processes in 
place to ensure that vacant beds are being identified 
on a timely basis and to improve communication of 
bed availability between in-patient units and emer- 
gency departments. One of the hospitals has been 
able to capture the times at which beds are cleaned. 
Another hospital has a central bed allocation team 
to manage an automated system that provides a 
real-time report to identify vacant beds that are 
ready to receive patients from the emergency 
department. The third hospital is in the beginning 
phase of implementing a new technology that will 
provide real-time direct communication across the 
hospital of information including discharges and 
changeover of beds, and the need for cleaning. 
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The Ministry informed us that the Emergency 
Department Process Improvement Program 
(ED PIP) and the Pay-for-Results program 
have focused on reducing unnecessarily long 
stays of admitted patients in emergency 
departments. Specifically: 

• The key focus of the ED PIP is to improve 
patient flow through engaging housekeepers, 
nurses and porters to develop a streamlined 
process and to reduce the overall bed- 
turnaround time ("bed-empty time") . The 
Ministry has engaged the Institute for Clinical 
Evaluative Sciences (ICES) to evaluate the 
effectiveness of the ED PIP According to 

the preliminary results of the evaluation, 
the hospitals that participated in the ED PIP 
have improved their emergency-department 
wait times. The final report is expected in 
March 2013. 

• As part of the Pay-for-Results program, 
the Ministry has provided 24 emergency 
departments with dedicated funding of about 
$26 million in the 2011/12 fiscal year to 
create and operate 212 short-stay in-patient 
beds in order to place admitted patients from 
emergency departments quickly in in-patient 
units. The results from May 2012 showed 
that the time patients waited in emergency 
departments for transfer to in-patient units 
was 21 hours, a reduction of 11% from the 
23.6 hours patients waited in May 2011. 

STAFFING 

Recommendation 6 

To ensure that emergency departments are operating 
cost-effectively with adequate nurses and physicians: 

• hospitals should deal with chronic overtime by 
setting targets for reducing overtime costs to 
acceptable levels and implementing effective 
measures for achieving these targets; and 

• the Ministry of Health and Long-Term Care 
should work with the LHINs and with hospitals 
to conduct studies to assess the requirements, 



availability, and regional distribution of 
emergency physicians across the province in 
order to develop a sustainable human resources 
strategy that will ultimately eliminate the use 
of agency physicians. 

Status 

Two of the hospitals we visited indicated that 
overtime has continued to be an issue because 
recruiting and retaining skilled emergency- 
department nurses have continued to be a challenge. 
All three hospitals we visited have implemented 
some measures to reduce overtime costs. Two of the 
hospitals have reviewed staff schedules as well as 
trends of maternity leaves, sick days, vacations and 
overtime to ensure appropriate staffing patterns 
and to keep ahead of possible vacancies, thereby 
reducing the need for overtime. Another hospital has 
made reducing overtime a corporate goal, and its 
emergency department has had some success over 
the past two years in reducing overtime hours as a 
percentage of total worked hours from 6.6% in the 
2008/09 fiscal year to 5.8% in 2010/11. It is also in 
the process of finalizing a recruitment and retention 
strategy for implementation in 2012/13. 

The Ministry informed us that it has introduced 
several initiatives to ensure that an adequate 
number of physicians are available in emergency 
departments. For example: 

• The Ministry offered the Summer Incentive 
for Designated Emergency Departments 

in 2011 to provide additional emergency- 
physician coverage and to keep emergency 
departments open during the challenging 
summer months. Up to $2.1 million was 
provided to help keep 97 emergency depart- 
ments open during the summer of 2011. The 
incentive is being continued in 2012. 

• The Ministry has provided one-time funding 
of about $2.5 million over the three fiscal 
years 2011/12 through 2013/14 for a pilot 
program, called the Supplemental Emergency 
Medicine Experience (SEME) program, to 
provide family physicians with three months 
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of full-time, remunerated continuing educa- 
tion and training in emergency departments. 
The University of Toronto has implemented 
the SEME program to enhance emergency- 
medicine skills for up to 20 family physicians 
a year and to improve coverage in small rural 
emergency departments. 

• The Ministry in collaboration with the 
Ontario College of Family Physicians, has 
offered an Emergency Medicine Primer 
for Family Physicians to reintegrate family 
physicians into emergency departments. 
This course has been offered in locations 
across Ontario as an update course for family 
physicians who are currently working in an 
emergency department and as a refresher 
course for family physicians who have worked 
in emergency departments before. 

IMPACT OF EMERGENCY-DEPARTMENT 
WAIT TIMES ON AMBULANCE 
EMERGENCY MEDICAL SERVICES (EMS) 

Recommendation 7 

To ensure the efficient use of the ambulance 
Emergency Medical Services (EMS) and to enhance 
co-ordination between EMS providers and emergency 
departments, the Ministry of Health and Long-Term 
Care should: 

• determine whether the recommendation in 
the 2005 expert panel's report on ambulance 
effectiveness of a benchmark ambulance offload 
time of 30 minutes 90% of the time should be 
accepted as a province-wide target; 

• work with hospitals, EMS providers, and Cancer 
Care Ontario to improve the validity and reli- 
ability of ambulance offload data and to ensure 
that such data are standardized, consistent, and 
comparable; and 

• work with hospitals and EMS providers to evalu- 
ate on a province-wide basis the effectiveness of 
the Offload Nurse Program in reducing offload 
delays and improving patient flow within 
emergency departments. 



Status 

The Ministry informed us that the Pay-for-Results 
program has included a benchmark for ambulance 
offload time of 30 minutes 90% of the time. The 
74 hospitals that have participated in the Pay-for- 
Results program are required to submit ambulance 
offload data and to demonstrate their progress 
toward meeting the 30-minute target. 

The current challenge, according to the Ministry, 
is the lack of a provincial definition of Ambulance 
Transfer of Care. To address this, the Ministry has 
been working with the hospitals, EMS and CCO 
on data collection by using the current National 
Ambulatory Care Reporting System definition 
for Ambulance Transfer of Care as a provincial 
definition for both hospitals and EMS. 

The Ministry introduced the Offload Nurse 
Program in 2008/09 by providing $4.5 million 
to 14 municipalities experiencing offload delays, 
creating 42 offload nurse positions. In the 2010/11 
fiscal year, the Ministry expanded the program 
to 16 municipalities and increased the funding to 
$9.6 million, providing about 191,000 additional 
nursing hours. In 2011/12, the Ministry further 
expanded the program to 18 municipalities with 
total funding of $11.7 million to provide about 
225,400 additional nursing hours. According to 
the evaluation by the Ministry, the municipalities 
have reported improvements in ambulance offload 
delays, as the Offload Nurse Program has freed up 
six fully staffed ambulances per day to respond to 
new urgent calls in a timely manner. 

All three hospitals we visited informed us 
that they have been working closely with their 
respective LHINs and other hospitals in monitoring 
ambulance offload time. They have also compared 
their results with peer hospitals within their 
regions and across the province. As well, they have 
continued to collaborate with the Ministry and the 
EMS in their regions to evaluate the effectiveness of 
the Offload Nurse Program in reducing ambulance 
offload time. 
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PERFORMANCE MONITORING 

Recommendation 8 

To ensure that emergency departments are provid- 
ing high-quality emergency care to all patients, 
hospitals should: 

• promote a culture of patient safety by using a 
non-punitive and "lesson-learned" approach to 
ensure that adverse events are reported and sum- 
marized for analysis and corrective actions; and 

• follow up with patients who have been triaged 
as having serious medical conditions but who 
have left emergency departments without being 
seen by doctors or having completed treatment. 

Status 

All three hospitals we visited advised us of actions 
taken to promote a culture of patient safety. One 
of the hospitals has a newly implemented Critical 
Incident Process. Another hospital introduced an 
automated incident-reporting system called SAFE 
in June 2010 and has a separate review process 
for critical incidents. The third hospital has imple- 
mented a Good Catch Campaign and an online 
incident-reporting process. 

To protect and promote a culture of quality 
improvement in hospitals, the Ministry also 
informed us that the government has made several 
legislative changes. For example: 

• According to the Excellent Care for All Act, 
2010, all hospitals are required to develop 
annual quality improvement plans that 
include critical incident data. Hospital 
boards are required to ensure that hospital 
administrators have established a system for 
analyzing critical incidents and developing 
plans to avoid or reduce the risk of further 
similar incidents. 

• As of January 1, 2011, Regulation 965 of the 
Public Hospitals Act was amended to ensure 
that hospital administrators provide aggre- 
gated critical incident data to their hospitals' 
Quality of Care Committees at least two 
times per year. 



• The Ministry issued a directive that as of 
October 1, 2011, all public hospitals are 
required to report all critical incidents 
related to medication and IV fluids through 
the National System of Incident Reporting 
(NSIR) within 30 days following the disclo- 
sure of the critical incident to the Ministry's 
Medical Advisory Committee, the hospital 
administrator and/or the patient. 

• The 2011 Better Tomorrow for Ontario Act 
has created an exemption under the Freedom 
of Information and Protection of Privacy Act 
that would give hospitals the discretion to 
refuse to disclose records in cases where 
people providing the information have 

an expectation of confidentiality. The 
exemption, effective since January 1, 2012, 
encourages hospital staff to engage in full, 
free and frank discussions on quality-of-care 
issues without fear that their comments will 
be publicly revealed. 
All three hospitals we visited have procedures in 
place to deal with patients who have left emergency 
departments without having being seen by doctors 
or having completed treatment. Two of the hospi- 
tals have required their nurses or the manager of 
the emergency department to follow up by phoning 
those patients. As well, all three hospitals have 
shown improvements, with fewer patients leaving 
their emergency departments without being seen, 
as a result of the ED PIP and the Pay-for-Results 
program. One hospital has reduced its percentage 
of patients who leave without being seen from 4% 
to 0.4%; another hospital has reduced the percent- 
age from 6.1% to 3%; and the third hospital has 
reduced the percentage from 5.4% to 0.5%. 

ALTERNATIVES TO EMERGENCY- 
DEPARTMENT SERVICES 

Recommendation 9 

To ensure that the needs of patients are met 
appropriately, the Ministry of Health and Long-Term 
Care should: 
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• work with hospitals to conduct further research 
on the impact of low-acuity patients on 
emergency services and on what province-wide 
initiatives can be undertaken to encourage 
people to seek the right treatment from the right 
medical provider; and 

• assess and promote the availability and 
public awareness of health-care alternatives 
to emergency departments on a regional basis, 
including walk-in clinics, urgent care centres, 
family physicians, and other community-based 
supports, to optimize the right care in the right 
environment. 

Status 

The Ministry informed us that it has continued its 
efforts to increase public awareness of alternative 
services to emergency departments in order to 
encourage people to seek the right treatment from 
the right health-service provider. For example: 

• In November 2010, the Ministry re-launched 
the Your Health Care Options website, which 
includes an online registry for Health Care 
Connect to help any Ontarian who is without 
a family doctor to find one, and searchable 
tools to obtain information on wait times 
for surgeries and diagnostic services. From 
May 2011 to August 2011, the Ministry 
launched a campaign for the Your Health Care 
Options website through television, radio, 
online advertising and materials distributed to 
health-care providers. 

• In May 2011, the Ministry deployed 20 kiosks 
at selected emergency departments to provide 
users with access to the Your Health Care 
Options website and a directory where they 



can search for health-service providers in their 
community as an alternative to the hospital 
emergency department. All 20 kiosks were 
implemented by August 2011, and users had 
made over 220,000 searches as of April 2012. 
• In the 2011/12 fiscal year, the Ministry 
provided $3.5 million as base funding for 42 
full-time nursing positions for the Nurse-Led 
Outreach Teams (NLOTs), which provide care 
directly to long-term-care-home residents 
to avoid unnecessary visits to emergency 
departments. Supplemental funding of about 
$1.5 million was provided to support the 
creation of 30 additional full-time nursing 
positions to augment existing NLOTs in 
selected LHINs. 
At the time of our follow-up, ministry data 
showed that these efforts had reduced the 
volume of low-acuity patients visiting emergency 
departments as a percentage of total emergency- 
department visits from 45% in 2009/10 to 41% in 
2011/12. 

Apart from the Ministry's efforts, all three hos- 
pitals we visited have also taken actions. One of the 
hospitals informed us that it opened an urgent care 
centre in April 2011 and has worked with its public 
relations staff to educate the community regarding 
the appropriate use of the emergency department 
and the urgent care centre. Two of the hospitals 
have worked with their stakeholders, including 
LHINs, peer hospitals and long-term-care homes, 
to promote the use of alternative services such as 
out-patient clinics in order to reduce emergency- 
department visits and hospital admissions. 
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Over the decade from the 2000/01 to 2009/10 
fiscal years, the Ministry of Training, Colleges and 
Universities (Ministry) provided Ontario's 24 col- 
leges of applied arts and technology with facility 
renewal funding totalling $13.3 million a year, 
supplemented by periodic additional allocations 
for renewals totalling $270 million. The combined 
annual and additional funding amounted to 
$403 million over the 10-year period. 

In addition to funding facility maintenance, the 
Ministry also provides capital grants to enhance 
and expand the colleges' capital needs. In recent 
years, the Ministry has provided this funding 
primarily for new facilities space so that colleges 
can accept more students. In 2009, the federal 
government initiated the Knowledge Infrastructure 
Program (KIP), a two-year infrastructure program 
for Canadian colleges and universities. 

The federal and provincial governments together 
provided capital grants to colleges for new facili- 
ties totalling $300.5 million between the 2006/07 
and 2009/10 fiscal years to create local short-term 
employment and increase the number of students 
that colleges could accommodate. Facilities space 
includes classrooms, laboratories, cafeterias, offices, 
libraries, and other such required areas. 



Our 2010 audit focused on the adequacy of 
endeavours by the Ministry and selected colleges 
to maintain college assets in a good state of repair. 
Along with our work at the Ministry, we visited 
the following colleges: Algonquin and La Cite in 
Ottawa; Confederation in Thunder Bay; and George 
Brown and Humber in Toronto. We also contacted 
six other colleges to obtain their input on specific 
issues and met with various stakeholders, includ- 
ing Colleges Ontario and the Council of Ontario 
Universities. 

Our work indicated that although colleges have 
benefitted from the new-facility capital funding, 
ongoing funding for maintenance of existing facili- 
ties had not been sufficient to maintain the aging 
college infrastructure, and the backlog of deferred 
maintenance was increasing. Some of our more 
significant observations were: 

• The Ministry was in the process of imple- 
menting a long-term capital planning process 
but did not have a formal plan in place at the 
time of our audit for overseeing the colleges' 
investment in infrastructure. 

• Many colleges had not maintained adequate 
asset management systems to facilitate 
effective capital planning and performance 
reporting on the condition and use of their 
capital infrastructure. 
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Status of Actions Taken on 
Recommendations 



• As of April 2010, the deferred maintenance 
backlog — the cost to perform all needed main- 
tenance and repairs — exceeded $500 million 
and had been increasing annually. More than 
$70 million in capital repairs were in the 
critical category and needed to be dealt with 
in the next year. 

• As of April 2010, about half of the college 
system's infrastructure assets were likely in 
poor condition when rated according to a rec- 
ognized industry standard that measures the 
state of infrastructure. 

• Applying the funding guideline of 1.5% to 
2.5% of asset replacement cost outlined by the 
U.S. -based Association of Higher Education 
Facilities Officers, annual ministry funding 

to all colleges over the 2006/07-2009/10 
fiscal years would have needed to be in the 
$80 million to $135 million range. However, 
actual capital renewal funding has remained 
at $13.3 million annually for many years; 
even with the periodic additional funding of 
$270 million, the total barely adds up to half 
of this guideline amount. 

• Administrators at all of the colleges we visited 
indicated they had to supplement ministry 
renewal funds with operating funds to help 
address their most urgent priorities or run the 
risk of assets deteriorating prematurely. 

• Until very recently, ministry funding decisions 
often lacked transparency and consistent 
criteria to evaluate funding requests, and 
there was insufficient documentation to dem- 
onstrate compliance with eligibility criteria. 

We made a number of recommendations for 
improvement and received commitments from 
the Ministry and the colleges that they would take 
action to address our concerns. 



According to information provided by the Ministry 
and the five colleges we visited as part of our 2010 
audit, some progress has been made on imple- 
menting the recommendations from our 2010 
Annual Report, with substantial progress on a few. 
The Ministry informed us that it has improved its 
capital-project selection process and is continuing 
to develop its capital planning process for colleges. 
In addition, as a prerequisite for infrastructure 
funding, every college will be required to develop 
an asset management plan that will outline the 
condition of its existing assets and the institution's 
plan for addressing its renewal needs. To this 
end, the Ministry has initiated projects to create 
a comprehensive college facility space inventory 
and establish utilization standards, and complete a 
system-wide assessment of the condition of college 
facilities across the province. Establishing space 
standards and assessing the condition of facilities 
are key first steps in developing a sustainable long- 
term asset management plan, including priorities 
for addressing deferred maintenance. As this work 
is expected to be completed in two years, deferred 
maintenance will remain a significant issue in the 
foreseeable future. 

CAPITAL PLANNING 

Recommendation 1 

To help ensure that capital infrastructure grants are 
allocated on the basis of clearly identified needs and 
province-wide priorities, the Ministry of Training, 
Colleges and Universities needs to continue developing 
a formal long-term capital planning process using 
current and reliable information obtained from the 
colleges and make funding decisions based on more 
predictable, rigorous, and clear criteria. 
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Status 

The Ministry informed us that it is continuing 
to develop its capital planning process, and in 
May 2010 sent out a memo to all colleges asking 
them to identify their capital planning priorities. 
The memo requested that each college provide 
an update of its inventory of capital projects and 
major strategic infrastructure initiatives, includ- 
ing asset management strategies and a ranking of 
project proposals. Ministry priorities for projects 
included criteria such as the capacity for growth 
in high-demand areas, a contribution to long-term 
economic development, and demonstrated linkages 
with provincial priorities in post-secondary educa- 
tion, including growth in the college sector and the 
provision of quality education. 

The Ministry has developed a project-selection 
framework to assess all project proposals received 
from the colleges with an approach focused on 
assessing the alignment of the proposals with its 
own priorities and those of the Ministry of Infra- 
structure. In June 2011, the Ministry selected 10 
capital projects at colleges and committed funding 
of $245 million with an estimated incremental 
enrolment of more than 9,200 students. 

In May 2011, the government announced "Put- 
ting Students First: Ontario's Plan for Postsecond- 
ary Education." The plan is designed to support 
enrolment growth by providing space and operat- 
ing funding for 60,000 more students at colleges 
and universities over the next five years. The plan 
also emphasizes strategic and well-planned growth 
in the post-secondary education sector. 

In June 2011, the Ministry of Infrastructure 
released Ontario's long-term infrastructure plan, 
Building Together: Jobs and Prosperity for Ontarians, 
which outlines the government's plans for the next 
10 years. The plan notes that Ontario will work with 
colleges and other post-secondary institutions to: 
• ensure that infrastructure investments 
respond to demand, align with the aims of 
"Putting Students First" and support the 
goal of a 70% attainment rate for post- 
secondary education; 



• introduce a satellite campus policy to help 
manage growth in the system and give prior- 
ity to areas where rapid growth is expected; 

• develop a more comprehensive funding 
policy for major capital projects, including 
procurement approaches and a framework 
for determining the appropriate provincial 
share; and 

• emphasize asset management planning and 
facilities renewal and repurposing. 

As a prerequisite for infrastructure funding, 
every college and university will be required to 
develop an asset management plan that outlines 
the condition of its existing assets and its plan for 
addressing the renewal needs identified. When 
seeking funding for expansion, institutions will 
need to provide a clear rationale to opt for new 
construction instead of renewing or repurposing 
existing space. 

In collaboration with the Ministry of Infrastruc- 
ture, the Ministry intends to ensure that institutions 
comply with the new requirements set out in the 
provincial 10-year infrastructure plan, including 
developing and publishing detailed institutional 
asset management plans as a prerequisite to receiv- 
ing provincial capital funding. The Ministry has 
also initiated projects to create a comprehensive 
college facility space inventory, develop space stan- 
dards and assess the condition of facilities across 
the province. 

The colleges informed us that the Ministry had 
asked them to submit updates to their proposed cap- 
ital projects in 2008 and 2010, but there had been 
no further requests since then. One college noted 
that these requests represent an excellent founda- 
tion for a possible province-wide capital planning 
process. However, the same college also expressed 
concern that the process needed to be ongoing 
in order to provide current and reliable planning 
information. Another college noted that the Min- 
istry promised to continue consulting with colleges 
regarding asset management planning and that it 
subsequently had a number of meetings with min- 
istry staff regarding growth and facilities planning. 
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FACILITIES RENEWAL AND 
MAINTENANCE 

Recommendation 2 

To preserve the taxpayer's investment in the college 
infrastructure and maintain these assets in good 
condition so that colleges can provide an adequate 
learning environment, the Ministry of Training, Col- 
leges and Universities should continue to work with 
Ontario colleges to: 

• ensure that the asset management information 
system is regularly and consistently maintained 
to enable both the Ministry and colleges to make 
informed decisions based on current, accurate, 
and complete information; and 

• develop strategies, targets, and timelines to 
address the deferred maintenance backlog. 

Status 

The Ministry's facilities renewal program is 
intended to assist post-secondary institutions in 
addressing ongoing maintenance, repairs and 
renovations of existing facilities. Recurring annual 
facilities renewal funding had been $13.3 million 
over the previous 10 years and was provided 
to supplement the colleges' own annual capital 
maintenance programs. Ministry funding for 
capital maintenance decreased in 2011/12 and 
is not expected to increase, as we were informed 
that funding for facilities renewal was budgeted at 
$8.7 million for the 2012/13 fiscal year. Estimates 
provided by the colleges indicate that components 
of the 10 approved major capital projects (totalling 
$245 million) are projected to contribute an addi- 
tional $14 million to address the colleges' current 
deferred maintenance backlog. 

The Ministry and colleges are working to 
develop an inventory of college spaces that support 
student enrolment, as well as to develop related 
standards, consistently assess facility conditions 
across the college sector and determine the appro- 
priate priorities for addressing deferred mainten- 
ance given available funding. Establishing space 
standards and assessing the condition of facilities 



are key first steps to developing a sustainable long- 
term asset management plan. 

Ontario colleges currently maintain their space 
inventory data in a variety of formats. Further- 
more, there is no common definition for different 
categories of college student space, nor are there 
standards for the amount of space required for each 
category. To address this issue, the Ministry and 
Colleges Ontario (the advocacy organization for the 
province's colleges of applied arts and technology) 
signed a transfer-payment agreement in October 
2011 to work on the development of system-wide 
space standards and utilization benchmarks, and to 
complete a system-wide space inventory. 

In November 2011, Colleges Ontario engaged 
a consultant to lead the work to develop a space 
inventory, utilization and planning framework for 
Ontario's 24 colleges. Key deliverables stipulated in 
the agreement include completion of a system-wide 
space inventory by September 2012; implementa- 
tion of a common space inventory database; and 
development of standards, benchmarks and a 
reporting system for space utilization. On Febru- 
ary 27, 2012, the Ministry received a progress 
report on the work accomplished to date, which 
largely focused on the development of a common 
space inventory database. At the time of our follow- 
up, the Ministry was expecting another progress 
report on the space inventory and database by 
September 15, 2012. Colleges Ontario was planning 
to deliver its final report by January 31, 2013. 

In April 2012, the Ministry received approval 
to procure a vendor to refine and standardize the 
existing facility condition assessment systems in 
use by Ontario's colleges. To ensure that the results 
of the procurement meet the needs of all decision- 
makers, a working group with representatives 
from the Ministry, the colleges, the Ministry of 
Infrastructure and Colleges Ontario was set up to 
provide advice. The project is expected to ensure 
consistent standards in key areas such as replace- 
ment values and unit costs, and to undertake 
system-wide facility condition assessments with 
related data updates. The project will be funded by 
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the Ministry and is expected to be completed within 
two years. At that time, consideration will be given 
to initiating a cycle of updates to the assessment for 
up to 20% of facilities per year. 

Establishing space standards and assessing 
the condition of facilities provide the information 
needed to make good investment decisions. Once 
better information on the demand for space and 
the condition of existing space is available, updated 
estimates on the funding required to meet identi- 
fied needs can be prepared. 

Several of the colleges advised us that they were 
actively participating in the Ministry's assessment 
of space utilization and/or facility condition. One 
college noted that consistently updating the space 
inventory and utilization data for all colleges was 
an excellent step toward effective asset manage- 
ment, and it anticipated that all colleges would be 
up to date by the end of March 2013. The college 
also noted that, since no system-wide strategies 
appeared to be emerging and there was no new 
money and even less facilities renewal funding, it 
is continuing to develop its own strategies to cope 
with its deferred maintenance backlog. This often 
results in addressing only the most urgent needs. 

MAJOR CAPITAL PROJECT MANAGEMENT 

Recommendation 3 

To help ensure that new construction and major 
renovations efficiently and cost-effectively achieve 
both college capacity goals and ministry economic 
objectives, the Ministry of Training, Colleges and 
Universities should: 

• implement fair and transparent procedures, 
similar to those developed for the Knowledge 
Infrastructure Program, for its project proposal, 
evaluation, and selection process; 

• enter into an agreement with each college to 
indicate the Ministry's and college's respective 
responsibilities for completing the project and 
the necessary reporting requirements; 

• advance funds to colleges as the work pro- 
gresses; and 



• maintain adequate documentation throughout 
the process to demonstrate that the program is 
transparent, fair, and achieves value for money, 
as well as college and ministry objectives. 

Status 

In May 2010, the Ministry requested that colleges 
identify their capital planning priorities and pro- 
vide an update to their inventory of capital projects 
and major strategic infrastructure initiatives, 
including institutions' asset management strat- 
egies and a ranking of their project proposals. To 
assess all project proposals received, the Ministry 
developed a project selection framework with an 
approach focused on assessing the alignment of 
the proposals with overall provincial infrastructure 
priorities and initiatives, as well as their fit with 
the strategic priorities of Ontario's post-secondary 
education sector. 

Project assessment criteria included the esti- 
mated total and provincial cost (including cost per 
each new student enrolled); the type of project 
(for example, renewal, expansion); the impact on 
capacity growth in underserved regions; align- 
ment with key government initiatives, such as the 
province's growth plans; and links to key sectors 
outlined in Ontario's innovation agenda (such 
as health services, science and technology, and 
skilled trades). 

These project assessment criteria were included 
in the evaluation templates that ministry staff com- 
pleted in order to evaluate all the project proposals 
received from the institutions. After the initial 
project assessment, ministry staff met with repre- 
sentatives from the colleges to review their project 
proposals and their alignment with institutional 
priorities and ministry capital planning objectives. 
Following these meetings, the Ministry identified 
10 capital projects for funding. 

The Ministry entered into transfer payment 
agreements with all institutions whose infra- 
structure projects were selected, with the excep- 
tion of three large and complex infrastructure 
projects approved for Alternative Financing and 
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Procurement (AFP), a construction delivery 
model developed and managed by Infrastructure 
Ontario. The Ministry will enter into transfer pay- 
ment agreements for the AFP projects once the 
projects are at the appropriate stage. No funding 
will be flowed to these three institutions until 
agreements are in place. The Ministry's transfer 
payment agreements stipulate that each institution 
is required to comply with the Procurement Direc- 
tive that came into effect on April 1, 2011, under 
the Broader Public Sector Accountability Act, 2010. 
The agreements were reviewed and approved by 
the Ministry of the Attorney General to ensure they 
protect, to an appropriate degree, the interests of 
the province. 

The Ministry's project oversight includes 
monitoring progress on the projects and ensuring 
compliance with the government's Transfer Pay- 
ment Accountability Directive and with the Cash 
Management Directive. The latter directive requires 
payments be made at or close to the time money is 
needed by the recipient. As part of the process, in 
addition to the pre-existing annual and close-out 
reporting requirements, the Ministry has imple- 
mented a quarterly reporting cycle that captures 
expenditure reporting and provides details that 
enable the Ministry to monitor project progress 
and provide transfer payments as the construction 
phase progresses. 

The Ministry also introduced a new requirement 
for institutions to provide a cost consultant's report 
reviewing the projected costs of the projects. In 
addition to the pre-existing requirement that each 
institution provide a copy of the Certificate of Sub- 
stantial Performance in accordance with subsection 
2(1) of the Construction Lien Act upon completion 
of the project, colleges are now required to provide 
a commissioning agent's report to confirm that the 
building systems work as they are intended to. 

The Ministry intends to refine its existing project 
selection and project oversight to reflect the long- 
term capital planning objectives and provincial 
fiscal realities. Given limited capital funding flex- 



ibility, the Ministry has not issued a request to col- 
leges for updated or new proposals since 2010. 

With respect to submissions, project selection, 
funding agreements, cash flow and monitoring, 
one of the colleges responded that there has been 
a noticeable increase in rigour in the requirements 
related to the documentation and reporting for its 
funded capital projects. The college also noted that 
these projects still necessitated a significant amount 
of effort to solicit support for approvals, and it 
would like to see a process that would help provide 
more predictable funding decisions. 

MEASURING AND REPORTING ON 
PROGRAM EFFECTIVENESS 

Recommendation 4 

To help ensure that all stakeholders have a good 
understanding of the condition of the province's college 
infrastructure assets, the Ministry of Training, Colleges 
and Universities and the colleges should continue to 
develop and report long-term performance indicators 
on the management and condition of their facilities. 

Status 

The Ministry is engaged in ongoing discussions 
with the sector on the data requirements for the 
asset management framework, including defining 
the most suitable asset management performance 
indicators. The outcomes from the space standards 
and facility condition assessment projects, to be 
completed over the next two years, will guide the 
development of performance indicators. 

In December 2011, the Ministry established 
the Deferred Maintenance Working Group with 
representation from the Ministry of Infrastructure, 
Colleges Ontario and various colleges. The Work- 
ing Group had a general discussion of the capital- 
related decision-making processes of colleges, 
the Ministry and the Ministry of Infrastructure. 
The aim was to determine what sorts of data were 
required to improve decision-making at all levels. A 
second meeting focused on the facilities condition 
assessments, specific pieces of data that would 
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need to be collected and the mechanics of col- 
lecting such data. 

Before the facilities condition assessments begin, 
a meeting will be held with the working group and 
the consultant engaged to refine and standardize 
the existing facilities condition assessment systems 
in use by Ontario's colleges. Data requirements 
will be finalized at this meeting and performance 
indicators discussed. The practical experience of 
the successful consultant will be combined with the 
reporting needs of colleges and the provincial gov- 
ernment to arrive at a final suite of indicators. While 
the working group is currently focused on imple- 
menting the facility condition assessment project, 
the focus will shift to how best to use the resulting 
data when the project is complete. 

Several colleges indicated that the two major 
initiatives in progress, the space standards and 
facility condition assessment projects, will contrib- 
ute to the development of performance indicators 
and reporting mechanisms to ensure consistent and 
comparable reporting for both the condition and 
utilization of college assets. 
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In January 2009, the federal government 
announced the Economic Action Plan, which 
included infrastructure investments, tax relief and 
grants to businesses and individuals as measures 
to stimulate the economy and combat the effects of 
the global economic crisis. 

The Plan also included several short-term pro- 
grams to support infrastructure projects and create 
jobs throughout 2009 and 2010. These programs 
targeted construction-ready projects that would not 
otherwise have been built within those two years, 
and had requirements that they be substantially 
completed by March 31, 2011. 

These programs were designed so that for every 
dollar that the eligible recipients — municipalities, 
First Nations and not-for-profit organizations — 
committed to an approved project, the federal 
and provincial governments would each commit 
another dollar. As well, a number of projects were 
undertaken by the province itself and funded 50/50 
with the federal government. It was estimated that 
with full take-up, the programs would lead to more 
than $8 billion in infrastructure spending across the 
province by the three levels of government. 

In 2010, we conducted an audit that focused 
on three of these programs, which accounted for 



about $3.9 billion of the total $6.9 billion federal- 
provincial short-term infrastructure commitment: 

• Infrastructure Stimulus Fund (ISF); 

• Building Canada Fund-Communities Com- 
ponent Top-Up (BCF-CC); and 

• Recreational Infrastructure Canada Program 
in Ontario and Ontario Recreational Program 
(RINC). 

The ISF and BCF-CC programs primarily sup- 
ported construction of roads, bridges, parks and 
trails, along with facilities such as municipal 
buildings and water and wastewater processing 
plants, while RINC helped build recreational infra- 
structure. When the two governments unveiled the 
programs in spring 2009, they set March 31, 2011, 
as the deadline for substantial completion of pro- 
jects. As of March 31, 2010, about $3.1 billion of the 
$3.9 billion available under the three programs had 
been committed to approximately 2,300 federal- 
provincial cost-shared projects. 

The Ontario Ministry of Energy and Infrastruc- 
ture (MEI — now divided into the ministries of 
Energy and Infrastructure) and the federal govern- 
ment were responsible for delivery of the three pro- 
grams. MEI was the lead ministry responsible for 
oversight and negotiating funding arrangements. 
On a day-to-day basis, the Ministry of Agriculture, 
Food and Rural Affairs administered ISF and BCF- 
CC, while the Ministry of Tourism and Culture 
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(now the Ministry of Tourism, Culture and Sport) 
administered RINC. 

In our 2010 Annual Report, we noted that in 
order to ensure that stimulus funds would be 
injected into the economy to create jobs as quickly 
as possible, the three programs were to give prior- 
ity to construction-ready projects of demonstrable 
benefit to their communities that could be substan- 
tially completed within two years. Priority was also 
to be given to those who planned to spend 50% or 
more of the funds by March 31, 2010, the end of 
the programs' first year. However, as of March 31, 
2010, the end of the first year of the two-year pro- 
grams, less than $510 million, or only about 16%, 
of the total $3.1 billion committed by the federal 
and Ontario governments had actually been spent. 
According to the job-creation model used by MEI, 
the three programs we examined would create and 
preserve about 44,000 jobs, but given the lower 
level of actual spending during the first year of the 
programs, only about 7,000 jobs were estimated to 
have been created or preserved during the first year 
of the two-year program. 

With respect to the grant-application and 
application-assessment processes, we noted that: 

• MEI placed no limit on the number of applica- 
tions that municipalities with populations 

of more than 100,000 could submit under 
ISF, the largest of the three infrastructure 
programs. This provided an incentive to 
submit large numbers of applications in hopes 
of getting as many approved as possible. For 
example, four municipalities submitted a total 
of almost 1,100 applications, accounting for 
40% of the total applications submitted by the 
421 Ontario municipalities for this program. 

• Due to the tight deadlines, the time allotted 
for the provincial review of ISF applications 
was in most cases just one to two days. In one 
instance, we noted that a key component of 
the provincial review for 56 projects worth 
an estimated $585 million was carried out in 
just four hours. In our view, it would not have 



been possible to conduct the necessary due 
diligence within such a tight time frame. 

• Applicants were not required to prioritize 
their infrastructure needs, and none did in 
their applications, making it more difficult to 
assess the benefits of the proposed projects 
so that informed funding decisions could be 
made. One municipality submitted 150 appli- 
cations valued at $408 million, and received 
approvals for 15 projects worth $194 million. 
From our visit to this municipality, we noted 
that 11 of the approved projects, valued at 
$121.7 million, were ranked at or near the 
bottom of the municipality's own priority list, 
while other, higher-ranked eligible projects 
were not approved. 

• We noted that technical experts were gener- 
ally not involved in assessing the applications 
even though thorough analysis by such 
experts would have helped assess the reason- 
ableness of project cost estimates and identify 
those unlikely to meet the two-year comple- 
tion deadline. 

After assessment and review by civil servants 
in the appropriate ministries, the applications 
were submitted to the office of Ontario's Minister 
of Energy and Infrastructure and to his federal 
counterpart for final review and approval. With 
respect to this process, we noted that there was a 
general lack of documentation to support the deci- 
sions regarding which projects were approved and 
which were not. In some cases, ministers' offices 
approved projects that civil servants had earlier 
deemed ineligible or about which they had flagged 
concerns. 

Federal and provincial funding was to end on 
March 31, 2011, after two years. As only 16% of 
the committed funds had been spent after the first 
year, many recipients had difficulty completing their 
projects by this deadline. Our survey indicated that 
as of May 2010, more than one-third of respondents 
had to take such steps as adjusting project specifica- 
tions and cost estimates in the original applications, 
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paying contractors overtime and sole-sourcing some 
contracts to meet the deadline. 

We made a number of recommendations for 
improvement and received commitments from the 
Ministry of Infrastructure (Ministry) that it would 
take action to address our concerns. 

In December 2010 (subsequent to our audit), 
both the federal and provincial governments 
announced a one-time extension of the March 31, 
2011, deadline until October 31, 2011, to provide 
one more construction season for projects not likely 
to meet the original deadline. In order to qualify 
for this new deadline, projects had to have incurred 
eligible costs before March 31, 2011, and were 
required to submit an application for the extension. 
According to the Ministry, nearly 800 projects were 
approved for this extension. In November 2011 and 
February 2012, the province exempted 21 projects 
from the extended deadline and continued to reim- 
burse its share of eligible costs incurred between 
November 1, 2011, and March 31, 2012. 

STANDING COMMITTEE ON PUBLIC 
ACCOUNTS 

The Standing Committee on Public Accounts held a 
hearing on this audit in March 2011. In May 2011, 
the Committee reported to the Legislature on this 
hearing. It stated in a letter that it was satisfied with 
the Ministry's responses to its questions and with 
the actions taken to date to address the Auditor 
General's recommendations. The Committee did 
not make any recommendations of its own. 



Status of Actions Taken on 
Recommendations 



The Ministry provided us with information in 
spring 2012 on the current status of the actions 
taken on our recommendations. According to this 
information, recommendations applicable to the 



2009 programs have been substantially imple- 
mented, in part because of the federal and provin- 
cial governments' extension of the funding deadline 
for qualifying projects to October 31, 2011, and the 
further provincial extension for qualifying projects 
to March 31, 2012. 

The Ministry agreed to take our recommen- 
dations pertaining to any future infrastructure 
programs into consideration when designing 
capital-grant programs. The status of the action 
taken on each of our recommendations at the time 
of our follow-up was as follows. 

PROGRAM ELIGIBILITY AND 
APPLICATION ASSESSMENT 

Program Design and the Submission of 
Applications 

Recommendation 1 

To help ensure that projects best suited to meeting pro- 
gram objectives are funded in any future infrastruc- 
ture programs, the Ministry of Infrastructure should: 

• follow a more risk-based approach to design- 
ing and implementing future capital-grant 
programs and consider all important factors 
affecting program delivery, including project 
suitability, reasonableness of timelines, and the 
capacity of and demand on ministry resources; 

• require that applicants better demonstrate the 
benefits of their proposed projects, provide evi- 
dence that the expected benefits are achievable, 
and prioritize their applications; and 

• strengthen its due-diligence process and include 
the use of technical experts to review high-risk 
projects, in assessing grant applications. 

Status 

Since our 2010 audit, the province has not initiated 
similar infrastructure programs. However, the Min- 
istry indicated that it is committed to taking this 
recommendation into consideration when design- 
ing any future capital-grant programs. Specifically, 
the Ministry informed us that it will expand the use 
of risk-based program design and analysis, assess 
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the implications of program-design decisions on 
ministry resources, and work to incorporate tech- 
nical expertise in the due-diligence process. The 
Ministry also committed to seeking the advice of 
the province's Internal Audit Division early in the 
development of future programs. 

For future infrastructure programs that involve 
partner ministries, the Ministry committed to 
providing better guidance to all staff responsible 
for reviewing applications, to strengthen the due- 
diligence process for application assessments. The 
Ministry also indicated that it will ensure that future 
application processes place a greater onus on appli- 
cants to demonstrate that the proposal and claims in 
their applications are valid and achievable and that 
they address a priority need for the community. 

Approvals of Applications 

Recommendation 2 

To ensure a fair and transparent project selection 
process is followed for any similar programs in future, 
the Ministry of Infrastructure should: 

• address all significant concerns raised during 
initial assessment and satisfactorily follow up 
and resolve them before approving the projects; 

• strengthen documentation of the rationale for 
decisions reached throughout all stages of the 
grant-assessment and approval processes; and 

• consider whether providing additional infor- 
mation would enhance transparency and 

be of interest to the general public and the 
Legislature. 

Status 

Similar to the previous recommendation, this rec- 
ommendation is also aimed at the administration 
of future capital-grant programs. However, after 
the March 31, 2011, deadline for the completion 
of projects under the 2009 infrastructure stimulus 
programs was extended to October 31, 2011, the 
Ministry made some efforts to implement the 
recommendation in the administration of the 
extended projects. For instance, recipients of the 



infrastructure stimulus funding were required to 
apply for the October 31, 2011, extension and to 
submit supporting documentation including a 
detailed construction schedule signed by a profes- 
sional engineer or architect. These applications 
were reviewed by the Ministry, the partner minis- 
tries and the federal government to assess whether 
each extension was compliant with the program 
criteria. The review process allowed for follow-up 
with applicants to seek clarification on issues and 
concerns identified in their applications before the 
decision on each extension was made. The Ministry 
maintained documentation to record whether 
each application for the extension was complete. 
In the case of incomplete applications, the reasons 
for their incomplete status were recorded so that 
the rationale for any denial of an extension was 
documented. Federal-provincial program manage- 
ment committees provided recommendations on 
extension approvals to the federal and provincial 
ministers, who had the final authority to approve 
extensions. 

To improve the transparency of projects funded 
by infrastructure stimulus spending, in December 
2010 the Ministry launched an enhanced version 
of the Revitalizing Ontario's Infrastructure website 
that contains additional information on the pro- 
jects, such as project descriptions and maps that 
better indicate the location of projects. In 2011, 
further enhancements were made to the website, 
which included improved project filtering and 
the addition of web pages showing the benefits 
provided by the province's infrastructure projects. 
The Ministry indicated that the website continues 
to be updated with information on the construction 
status of projects funded by stimulus spending, 
given that for some projects, payments for costs 
incurred up to March 31, 2012, will continue well 
into the 2012/13 fiscal year. 
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PROJECT MANAGEMENT 

Reporting and Monitoring the Progress of 
Projects 

Recommendation 3 

To help ensure that funded projects are completed 
on time and on budget, and to comply with funding 
agreements, the Ministry of Infrastructure should: 

• ensure that recipients report project informa- 
tion consistently and on a timely basis, and fol- 
low up on projects at risk of missing the funding 
cut-off deadline; and 

• consider raising the issue with the federal gov- 
ernment once reliable data is available on the 
number and extent of projects that will not be 
completed by the March 31, 2011, federal fund- 
ing cut-off 

Status 

In our 2010 Annual Report, we noted delays in 
reporting the status of 17% of the ISF municipal 
projects and 40% of the RINC projects. The status 
of the RINC projects, in particular, was difficult to 
determine reliably due to issues with RINC's infor- 
mation system. In June 2010, subsequent to our 
audit fieldwork, we were informed that progress 
reports had been submitted for 95% of all projects. 
However, we noted a variety of interpretations by 
recipients as to what constitutes "progress" on a 
project. Some defined it as estimated work done, 
while others used actual dollars spent and engin- 
eering assessments. The analysis we did in 2010 
of progress based on actual spending by recipients 
revealed that only 16% of the committed funds had 
been spent. This lagged significantly behind the 
province's initial expectation of equal spending in 
each of the program's two years and left a signifi- 
cant amount of the funds originally committed still 
to be spent in the 2010/11 fiscal year. 

As noted earlier, in December 2010, the federal 
government announced that it was extending the 
funding deadline for qualifying projects funded 
by stimulus spending to October 31, 2011, since 
these projects were at risk of not being completed 



by the March 31, 2011, cut-off date for federal 
funding. Close to 800 projects were approved for 
the extension. Subsequent to the extension by 
the federal government, the Ontario government 
further exempted 21 projects from the extended 
deadline and continued to reimburse its share 
of eligible costs incurred to March 31, 2012. The 
Ministry indicated that, as a result of these federal 
and provincial measures, only 19 projects had not 
been completed by March 31, 2012. According to 
the Ministry, most of these had incurred sufficient 
eligible costs prior to March 31, 2012, to have the 
full provincial contribution paid out. 

Costs to Meet the Funding Deadline 

Recommendation 4 

To help ensure that funds are spent wisely, the min- 
istries of Infrastructure, Tourism and Culture, and 
Agriculture, Food and Rural Affairs should work 
with any recipients experiencing significant delays on 
their projects to evaluate the options and solutions 
best suited to meet stimulus-program objectives and 
ensure value for money in completing the projects. 

Status 

The Ministry indicated that following our 2010 
audit, partner ministries engaged on an ongoing 
basis with recipients of stimulus spending, includ- 
ing conducting site visits to assess and validate 
project progress. The partner ministries regularly 
reported issues they noted to the Ministry and the 
federal government. They paid special attention 
to projects identified as delayed or otherwise at 
risk of not being completed by the new October 31, 
2011, deadline or the March 31, 2012, provincial 
deadline. The provincial ministries and the federal 
government collaborated to assess options and 
identify solutions for projects that needed support. 
This included working with recipients of stimulus 
funds to identify options such as changing the scope 
of their projects so that they could be completed by 
the funding deadline. 
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The Ministry also indicated that, ultimately, 
funding recipients were accountable under the 
binding stimulus funding agreement for the pro- 
curement, management and delivery of their own 
projects by the funding deadline. Follow-up letters 
approved by the Ministry, the partner ministries 
and the federal government were issued when war- 
ranted to ensure that recipients were aware of the 
current situation related to their projects and the 
potential consequences that could arise if they did 
not take the required action. 

As a result of the above actions by the Ministry, 
the partner ministries and the federal government, 
the number of projects considered at risk of not 
meeting the funding deadline steadily declined, 
and the vast majority of them were completed by 
March 31, 2012. 

Financial and Claims Administration 

Recommendation 5 

To ensure that funds are spent wisely and for the pur- 
pose intended, the Ministry of Infrastructure should 
work with the Internal Audit Division to develop 
appropriate monitoring and audit coverage of fund 
recipients according to assessed risk. 

Status 

In our 2010 Annual Report, this recommendation 
was predominantly aimed at the better monitoring 
of about 150 not-for-profit organizations (NPOs) 
that at the time of our audit had received approxi- 
mately $360 million in stimulus funding. Since 
these NPOs were typically smaller organizations 
that ranged from multi-purpose community organ- 
izations and recreational centres to special-purpose 
organizations like sports clubs, they might lack the 
project-management expertise and accountability 
structures of larger organizations. Furthermore, the 
provincial ministries involved with this program 
had no previous experience in dealing with these 
smaller organizations. 

Although since our 2010 audit it has not specific- 
ally audited any of the projects procured by these 



smaller NPOs using stimulus funding, the Ministry 
did indicate that a small sample of the statements 
of expenditures submitted by the NPOs was 
reviewed by an accounting firm to assess whether 
the statements were prepared in accordance with 
the terms of the funding agreement. The Ministry 
also indicated that it has been working with the 
Ontario Internal Audit Division (OIAD) to ensure 
that project and program close-out protocols are 
providing appropriate monitoring and risk mitiga- 
tion to achieve successful close-out of projects. 

The Ministry noted that the binding funding 
agreements with recipients give the province the 
right to audit projects to determine whether funds 
have been spent wisely. The province's right to 
audit extends beyond the completion of the project. 
The Ministry indicated that the government will 
continue to use the risk-assessment model to deter- 
mine if projects warrant an audit upon completion. 
To this end, following up on a recommendation pro- 
vided by the OIAD, the Ministry issued an Assistant 
Deputy Minister-level memo to partner ministries 
on April 3, 2012, about these ministries' ongoing 
accountability responsibilities as they relate to the 
stimulus programs. One such responsibility is to 
retain proper and accurate project and program 
records for six years following March 31, 2012, 
to support any audits that may be undertaken. 
The Ministry stated that it will continue to seek 
advice from the OIAD as the government continues 
to wind down the stimulus programs and in the 
design and implementation of future programs. 

MEASURING AND REPORTING ON 
PROGRAM EFFECTIVENESS 

Recommendation 6 

To better enable the public and legislators to evaluate 
the effectiveness of these stimulus programs, the Min- 
istry of Infrastructure should: 

• provide timely and accurate information on the 
progress of these projects; and 

• ensure that the methodology used to calculate 
the impact of stimulus funds on employment is 
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adjusted as needed to reflect the actual flow of 
funds into the economy and the impact on the 
job market. 

Status 

As noted earlier, to provide more timely and accur- 
ate information on the progress of projects funded 
by stimulus spending, shortly after our 2010 audit 
the Ministry added new features and additional 
information on the stimulus projects on its Revital- 
izing Ontario's Infrastructure website. 

The Ministry also indicated that it uses job 
multipliers provided by the Ministry of Finance 
(MOF), which MOF annually updates on the basis 
of the latest data from Statistics Canada. These 
updated multipliers, according to the Ministry, help 
ensure that the most current information about 
the structure of the Ontario economy is used in job 
calculations. 

Lastly, the Ministry stated that in view of the 
winding down of the stimulus programs, it con- 
tinues to work with its partner ministries and the 
federal government to identify the lessons learned 
from the stimulus programs so that future pro- 
grams can be made more effective and accountable. 
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Background 




The determination of the market value of a prop- 
erty is critical because it ultimately determines how 
much property tax an owner must pay. In Ontario, 
this tax is calculated by multiplying a property's 
assessed market value by the tax rate of the prop- 
erty's municipality. 

On December 31, 1998, the province transferred 
the responsibility for determining the assessed 
value for properties to the Ontario Property Assess- 
ment Corporation, later renamed the Municipal 
Property Assessment Corporation (Corporation). 
The primary responsibility of the Corporation's 
1,600 employees is to prepare an annual assess- 
ment roll for each local municipality that identifies 
each property and its assessed market value. In 
the 2011 calendar year, Corporation expenditures 
totalled $190.4 million ($185.5 million in 2009), 
most of which was funded by the province's 444 
municipalities. 

From the perspective of a property owner, it 
is reasonable to expect that each property will be 
assessed within a range that is reasonably close 
to its fair market value — the most likely sale price 
between a willing buyer and seller. At the time of 
our 2010 audit, this was also the position of the 
Corporation and Ontario's Assessment Review 



Board, the independent tribunal that hears appeals 
from people who believe their properties have been 
incorrectly assessed or classified. 

To get an indication of whether the Corpora- 
tion's mass-appraisal system achieved this object- 
ive, in our 2010 audit we compared the sale prices 
of 11,500 properties identified as having been sold 
at arm's length in 2007 and 2008 to their assessed 
market value as of January 1, 2008. We found 
that in 1,400 of these cases, or one in eight, the 
assessed value differed from the sale price by more 
than 20%. In many cases, the difference between 
assessed market value and actual selling price 
was substantial. 

At the time of our audit, the Corporation 
acknowledged that some individual property 
assessments might not have reflected the current 
or fair-market property-value range as indicated by 
an arm's-length sale price. These variations most 
often occurred because it did not have up-to-date 
property data from a property inspection. As a 
result, some property owners may have been over- 
or under-assessed, and therefore paid more or less 
than their fair share. While the Corporation did get 
it right for the vast majority of properties, this was 
of little solace to property owners who were over- 
assessed relative to neighbouring properties, and 
therefore paid more than their fair share of tax. 

More frequent property inspections and timely 
investigations of sales values that vary significantly 
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from assessed values should reduce the differences 
between assessed values and sale prices. Neverthe- 
less, our discussions with the Association of Muni- 
cipalities of Ontario indicated that municipalities 
were generally satisfied with the assessment-roll 
information the Corporation provided. 

In our 2010 Annual Report, we identified a num- 
ber of areas where improvement was needed with 
respect to the Corporation's collection of informa- 
tion essential for accurate and consistent property- 
tax assessments. The most significant of these areas 
were the following: 

• In the 1,400 cases in which we found the 
sale price differed by more than 20% from 
the assessed value, the Corporation had not 
investigated the reasons for these differences 
or made any adjustments to the assessed value 
of these properties where warranted. 

• We found almost 18,000 building permits 
with a total value of about $5.1 billion 

as of December 31, 2009, for which the 
Corporation had failed to inspect the cor- 
responding properties within the three-year 
statutory period for reassessing property 
and improvements. 

• Although the Corporation's target is to 
inspect each property in the province at 
least once every 12 years, under the actual 
inspection cycle it would take at least 18 years 
to achieve this, assuming current staffing 
levels and no further growth in the number of 
residential properties. 

• The Corporation began work on a new com- 
puter system in 2000, but the system was not 
yet fully functional, and costs incurred at the 
time of our audit exceeded $50 million com- 
pared to an original budget of $18.3 million. 

• While the Corporation had established rea- 
sonable requirements for acquiring goods and 
services, it often did not comply with good 
business practices, including its own manda- 
tory policies and procedures. 

We made a number of recommendations for 
improvement and received commitments from the 



Corporation that it would take action to address 
our concerns. 

STANDING COMMITTEE ON 
PUBLIC ACCOUNTS 

The Standing Committee on Public Accounts held a 
hearing on this audit in March 2011. In May 2011, 
the Committee tabled a report in the Legislature 
resulting from this hearing. The report contained 
five recommendations and requested that the Cor- 
poration report back to the Committee with respect 
to the following: 

• the Corporation's most recent data on 
significant variances between properties' sale 
prices and assessed values, including when 
and how the Corporation investigated the 
variances and what adjustments were made 
to assessments; 

• the status of the Corporation's efforts to 
obtain information on building-permit work 
completed and occupancy/completion status 
reports from municipalities, including the 
number of municipalities that are submitting 
this information and an indication of whether 
municipalities are reporting in a standardized 
way with enough detail and whether the Cor- 
poration would be publicizing on its website 
those municipalities with more than 5,000 
properties that: 

• have not submitted any information on 
building-permit work completed or any 
occupancy/completion status reports; 

• submitted building-permit information 
more than six months after work was 
completed; and 

• submitted occupancy/completion status 
information more than three months after 
the occupancy permit was issued; 

• the results of the Corporation's managerial 
file reviews of Requests for Reconsideration 
decisions since the audit; 

• the Corporation's progress in meeting a 
12-year inspection cycle, including the average 
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daily number of inspections per inspector 
(and the length of the ensuing inspection 
cycle), how the Corporation is ensuring that 
the risk of under- or over-assessments is the 
main driver of inspections, the effect of tools 
like hand-held computing devices, satellite 
imagery and phone surveys in shortening the 
inspection cycle, and measures taken to ensure 
high-quality field-office review of inspection 
files; and 

• what oversight procedures the Corporation 
has to ensure that its staff are complying with 
its revised policies on procurement and on 
travel, meals and hospitality. 
The Corporation formally responded to the 
Committee in September 2011. A number of 
issues raised by the Committee were similar to our 
observations. Where the Committee's recommen- 
dations are similar to ours, this follow-up includes 
the recent actions reported by the Corporation to 
address the concerns raised by both the Committee 
and our 2010 audit. 



Status of Actions Taken on 
Recommendations 



According to the information we received from 
the Corporation, significant progress has been 
made in addressing most of our recommendations, 
with some progress being made on the others. For 
instance, the Corporation has established a 25% 
threshold, above which differences between a 
property's sale price and its assessed market value 
are to be investigated within a 12-month period. 
The Corporation has also expanded its use of its 
information management systems to document 
property inspections and Requests for Reconsidera- 
tion. It also conducts managerial reviews on these 
electronic files. In addition, the Corporation has 
revised and strengthened its procurement policies 
to ensure that goods and services are acquired 
appropriately, written and properly authorized con- 



tracts are in place, and evaluations of contractors' 
performance are documented. It also advised us 
that it now follows the government's Travel, Meal 
and Hospitality Expenses Directive. The Corpora- 
tion will need additional time to fully address some 
of our other recommendations, such as inspecting 
and reassessing all properties for which a building 
permit has been issued before statutory limits on 
collecting additional taxes expire. The Corporation 
will also need additional time to implement its 
12-year residential property inspection plan. The 
status of the action taken on each recommendation 
at the time of our follow-up was as follows. 

ASSESSED VALUES OF 
RESIDENTIAL PROPERTIES 

Recommendation 1 

To help ensure that individual properties are assessed 
in accordance with the Assessment Act at the amount 
that a willing buyer would pay to a willing seller, the 
Municipal Property Assessment Corporation should: 

• formally establish a threshold above which 
differences between a property's sale price and 
its assessed market value must be investigated 
within a reasonable period of time; and 

• where warranted, adjust the property's assessed 
market value accordingly. 

Status 

The Corporation informed us that it used sale 
prices from 2009, 2010 and 2011 to develop values 
for its January I, 2012, assessment update. In 
2011, the Corporation established a threshold of 
25%, together with assessment-to-sales analysis 
from each market area, to flag sales that should be 
investigated. Depending on the difference between 
the previous assessed value and the current market 
value, the Corporation would conduct either an on- 
site inspection or a desktop review of the property. 
A desktop review, which is an alternative method 
to an on-site visit, consists primarily of reviewing 
digital images of the property and requesting sup- 
porting documentation from the property owner. 
The Corporation aimed to conduct all targeted 
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investigations within 12 months from the date it 
obtains sale information. 

The Corporation advised us that out of the 
approximately 1,118,000 sales that occurred 
between 2009 and 2011, it had identified about 
534,000 properties for investigations as they 
met the 25% threshold and a predetermined 
assessment-to-sales ratio. It had conducted inves- 
tigations on approximately 382,000 of them as of 
March 1, 2012. About 215,000 (or 56%) of these 
investigations resulted in a change to the property's 
assessed value. Approximately 191,000 (or 89%) 
were adjusted upward in value while the remaining 
24,000 (11%) were adjusted downward. In total, 
the exercise resulted in value changes of some 
$16.6 billion. The Corporation planned to complete 
the remaining approximately 152,000 investiga- 
tions by the end of 2012. 

The Corporation also informed us that it was in 
the process of updating its sales investigation pro- 
cedures, originally established in 2011. It planned 
to incorporate a risk-analysis model and develop 
additional alternatives to the procedures to ensure 
that sales investigations are completed on a more 
timely basis. 

BUILDING PERMITS 

Recommendation 2 

To help ensure that inspections of properties for which 
a building permit has been issued are completed on a 
timely basis so that retroactive assessments and tax 
can be levied as soon as possible and certainly before 
statutory limits expire, the Municipal Property Assess- 
ment Corporation should: 

• ask all municipalities in the province to provide 
the Corporation with formal notification when 
the work with respect to a building permit has 
been completed; and 

• inspect and reassess the market value of all such 
properties before statutory limits on collecting 
additional tax expire. 



Status 

The Corporation indicated that it has developed an 
extensive communications strategy and introduced 
various initiatives to encourage all municipalities 
to provide building-permit information via a 
standardized electronic process (in the absence of 
any legislative requirement to do so). It has also 
developed internal standardized reports, provided 
to its managers every month, that contain key 
building-permit information, such as the age of the 
permit, the actual or expected completion date for 
the project and the value of the permit issued. 

The Corporation advised us that the number of 
municipalities in compliance with the standardized 
process had more than doubled, from 69 to 154, 
over the four-month period from November 2011 
to February 2012. Although the 154 municipalities 
represent only 37% of the 414 municipalities that 
issue permits, these municipalities accounted for 
close to 126,000 (or 68%) of the approximately 
185,000 permits received by the Corporation for 
2010. The remaining 32% of permits (59,000) were 
submitted by 260 municipalities but were not pro- 
vided in compliance with the standardized process. 
The Corporation's goal was to have at least 80% 
of permits submitted in the standardized format. 
Once this 80% target is reached, the Corporation is 
to examine the costs versus the benefits of trying to 
raise this percentage further, as opposed to manag- 
ing the remaining paper permits manually. 

With respect to further inspection and reassess- 
ment of properties with a building permit issued, 
the Corporation informed us that it used the 
building-permit information received from muni- 
cipalities to prioritize certain properties for value 
assessment before statutory limits expire. The 
Corporation reported that this permit prioritization 
helped it to reduce the assessment amounts that 
had passed the statutory limits from 15.7% in 2010 
to 10% in 2011. The Corporation also indicated that 
the number of permits worth more than $10,000 
each that had been issued more than three years 
ago had been reduced from almost 18,000 as of 
December 31, 2009, to approximately 10,500 
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permits as of March 31, 2012. The total value of 
these older permits had also been reduced, from 
approximately $5.1 billion to about $2.3 billion over 
the same period. 

The Corporation further advised us that the 
average time to process an assessment change 
resulting from building-permit activity had been 
reduced from 10.8 months in 2010 to 9.6 months 
in 2011 and that the building-permit activity had 
resulted in over $28.4 billion in increased assess- 
ments in 2011. 

In addition, the Corporation indicated that, 
as of March 31, 2012, there were about 243,000 
building permits to be reviewed, including those 
issued in prior years. Of the 243,000 permits, about 
156,000 (or 64%) had been identified as ready for 
reassessment and prioritized for completion within 
legislated timelines. The construction work for the 
remaining approximately 87,000 (or 36%) building 
permits had not yet been completed. 

REQUESTS FOR RECONSIDERATION AND 
ASSESSMENT REVIEW BOARD APPEALS 

Recommendation 3 

To help ensure that the merits of Requests for Recon- 
sideration (RfRs) are properly assessed, and that the 
adjustments to the property's assessed market value 
are adequately supported, the Municipal Property 
Assessment Corporation should: 

• establish mandatory requirements for con- 
ducting and documenting RfRs; and 

• on a sample basis, conduct and document 
managerial file reviews of all RfRs, including 
those that result in no assessment changes, to 
ensure compliance with suggested requirements 
for conducting an RfR. 

Status 

The Corporation advised us that it has established 
and implemented new Requests for Reconsidera- 
tion (RfRs) Procedures, which identify the manda- 
tory requirements for conducting and documenting 
RfRs. These were implemented in October 2009 



and further revised in January 2011 to incorporate 
the new managerial review process. The Corpora- 
tion also implemented an automated document 
management system in 2010 to manage all RfRs. 
Process controls were incorporated into the docu- 
ment management system to improve documenta- 
tion quality and for ease of management review. 
Because all documentation and correspondence 
received from property owners is scanned and 
stored electronically, paper documents are no 
longer required. 

As of March 1, 2012, the Corporation had 
processed more than 18,500 RfRs received in 2011 
for that tax year. The Corporation indicated that 
manager reviews had been conducted for all of the 
2011 RfRs it received, including those that resulted 
in no change in assessment value. As a result of 
the managerial reviews, almost 2,700 (or 14.4%) 
of the 18,500 RfRs were flagged for further action. 
Of these, approximately 60% had errors relating to 
property valuation or classification. The remaining 
40% had inadequate supporting documentation, 
but there was no impact on the property value or 
owner. The Corporation informed us that all of 
these RfRs were further reviewed to ensure they 
had been accurately processed prior to issuing the 
final assessment for the property. 

The Corporation informed us that it will use 
the 14.4% rejection rate to determine the appropri- 
ate level of sampling for future managerial file 
reviews of RfRs. 

INSPECTIONS 

Property Inspection Cycle 

Recommendation 4 

To help ensure that the property information in its 
database is as complete and up to date as possible, 
and that it has reliable information with respect to 
inspections completed, the Municipal Property Assess- 
ment Corporation should: 

• require that each regional office select annually 
at least some properties for an inspection based 
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on the assessed risk of under- or over-assessment 
with a view to working toward meeting its 
12-year inspection cycle; and 
• maintain accurate and meaningful information 
with respect to the number and type of inspec- 
tions completed (for example, sales investiga- 
tions, building permits, and new constructions). 

Status 

The Corporation has established a 12-Year Property 
Review Plan and identified the number of proper- 
ties that must be reviewed each year in order to 
inspect all Ontario residential properties over a 
12-year period. The Corporation informed us that 
it has prioritized this listing to ensure properties 
deemed to be at risk of under- or over- assessment 
are inspected. Some of the at-risk properties were 
identified based on such factors as local area know- 
ledge, historical inquiries, appeals and RfR activ- 
ities, and the average time since the last inspection. 
The workload listing has been distributed to man- 
agers at each regional office bi-weekly since June 

2011. As of July 2012, the Corporation had com- 
pleted a total of approximately 206,000 property 
reviews for the year: approximately 111,000 were 
cyclical (targeted) inspections and approximately 
95,000 related to building permits, sales investiga- 
tions, RfRs or appeals. 

The Corporation indicated that the number of 
residential properties that have not been inspected 
or verified in more than 12 years had been reduced 
from over 1.5 million, as reported in our 2010 
AnnualReport, to about 1.3 million as of July 1, 

2012. Under its 12-Year Property Review Plan, the 
Corporation intends to gradually reduce the gap 
and eventually eliminate it by 2021. 

As was also reported in our 2010 Annual 
Report, not all the Corporation's inspections are 
conducted by an on-site visit. As mentioned earlier, 
the Corporation may validate property informa- 
tion through a desktop review depending on the 
assessed risk. The desktop review may include 
sending out sales questionnaires to property 
owners to obtain data for verification purposes. 



The Corporation informed us that, when changes 
identified by self-reporting property owners cannot 
be validated through the use of digital imagery, the 
validation is completed by way of an on-site review 
of the property. 

In addition, the Corporation informed us that 
to facilitate the tracking of accurate and meaning- 
ful information regarding the number and type 
of inspections completed, it added new property 
review reason codes and method codes in March 
2012 to improve its information systems. The 
new codes are intended to allow for more precise 
information that will better reflect the nature and 
outcome of the inspections. 

Inspector Workloads, Quality of 
Inspections Performed 

Recommendation 5 

To ensure that inspections are conducted efficiently 
and are adequately completed and documented, and 
support the changes to a property's assessed value, the 
Municipal Property Assessment Corporation should: 

• regularly monitor and assess the productivity 
of inspectors with respect to both the quality 
and average number of inspections being done 
each day; 

• ensure that files are documented in compliance 
with acceptable standards and clearly dem- 
onstrate what work was completed and what 
assessment changes were made as a result; and 

• oversee the success of each regional office in 
meeting the 12-year inspection-cycle target. 

Status 

The Corporation informed us that it has developed 
and, since September 2011, distributed monthly 
inspection reports that summarize work activities 
such as average inspections per day, total inspec- 
tions completed and total work time. The reports are 
intended to assist regional managers in their mon- 
itoring of inspector productivity and ensure that 
documentation standards are met; however, formal 
policies and procedures on the new documentation 
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standards had not yet been established. The Cor- 
poration planned to formalize the new requirements 
by early 2013. 

The Corporation also informed us that, effective 
June 2011, inspection results, such as inspection or 
verification dates, comments, outcomes and value 
changes (if any), are entered and stored in the 
Integrated Property System. Manual files comple- 
menting the system are no longer mandatory. 
Moreover, the Corporation indicated that property 
reviews were deemed to be completed only when 
the appropriate information had been updated in 
the system. 

With respect to meeting the 12-year provincial 
inspection-cycle target, the Corporation indicated 
that it had established a 12-Year Property Review 
Plan (see Figure 1) and an annual work plan at a 
field-office level, and it monitored progress on a 
monthly basis. The Corporation informed us that 
in 2011 it inspected approximately 543,000 proper- 
ties, or about 4,000 more than targeted. However, 
there are fluctuations in the inspection targets over 
the 12-year cycle related to such factors as existing 
workload, competing demands and lack of resour- 
ces. For example, because of the legislative require- 
ment for delivering the 2013 Assessment Roll by 



Figure 1: 12-Year Residential Property Review Cycle 

Source of data: Municipal Property Assessment Corporation 





# of Residential 


Year 


Property Reviews 


2010 


446,565 


2011 


538,500 


2012 


319,000 


2013 


261,929 


2014 


545,052 


2015 


566,245 


2016 


366,701 


2017 


314,315 


2018 


576,245 


2019 


580,908 


2020 


419,087 


2021 


314,315 


Total 


5,248,862 



December 2012, inspection targets for 2012 were 
reduced from the previous year by about 220,000 
(40%), to 319,000 inspections. 

We reported in 2010 that after the Ombudsman's 
2006 report recommended that the Corporation 
review its staffing needs, the total number of inspect- 
ors at the Corporation peaked at approximately 320 
in 2007; however, it had steadily dropped since then 
to about 230 as of April 2010. Although the Corpora- 
tion has increased its number of inspectors since, 
and had some 250 and 260 inspectors on staff as of 
December 2010 and 2011, respectively, it still has 
fewer staff than it did in 2007. 

In order to complete the 12-Year Property 
Review Plan using available resources, the Corpor- 
ation has introduced several initiatives to improve 
efficiency. For example, by the end of 2012, the 
Corporation planned to complete an upgrade of 
AboutMyProperty™, an on-line service property 
owners can use to validate their property infor- 
mation electronically. The Corporation believed 
this service would be used by significantly more 
property owners than in the past because of an 
upgrade making it much more accessible and user- 
friendly. Moreover, the Corporation has replaced 
traditional map books with GPS navigational 
devices and cellphones. As a result, its property 
inspectors have been able to locate properties 
more quickly and can get in touch with the office 
at any time for further information. 

Quality Control for Inspections Completed 

Recommendation 6 

To enhance the effectiveness of the current quality 
control function, the Municipal Property Assessment 
Corporation should: 

• ensure that supervisory reviews of inspection 
files are properly completed and adequately 
documented as required; and 

• include in its review process some inspection files 
that did not result in a change to a property's 
assessed value. 
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Status 

In July 2011, the Corporation introduced a Property 
Review Process Control Team to investigate oppor- 
tunities for streamlining and automating the prop- 
erty review process, to promote a consistent and 
more effective approach to management oversight, 
and to facilitate the assignment and reporting of 
random property reviews for each field office. 

Subsequently, in August 2011, the Corporation 
established standardized procedures to enable 
proper supervisory review of inspections. For 
example, the team developed a new form that 
is used to electronically document reviews on 
selected inspections on a bi-weekly basis. The Cor- 
poration informed us that property reviews were 
being done based on both the information in its 
Integrated Property System and on-site visits. The 
results of each review are then to be documented 
on the new form and distributed to the responsible 
management team for appropriate action and 
follow-up, if necessary. 

The Corporation informed us that about 44,600 
(or 8%) of the property reviews for 2011 were 
found to have problems, either because of non- 
compliance with standard operating procedures or 
because they were deemed to be duplicate reviews 
of the same properties. Although a property may 
have had to be reviewed more than once due to 
multiple work activities, for the purpose of contrib- 
uting to the Corporation's 12-Year Property Review 
Plan, the property was counted only once as a 
unique property review. 

The Corporation indicated that the samples 
selected for the review process included both 
properties where the initial inspection resulted in a 
change to either the property record or the assess- 
ment, and properties where the inspection resulted 
in no changes. In 2011, about 2,600 properties were 
sampled, of which approximately 1,500 (or 60%) 
were properties where the initial inspection had 
resulted in no change. As of December 31, 2011, the 
reviews of these 2,600 properties had resulted in 
absolute value changes of 0.3% of their total value. 



The Corporation informed us that it was con- 
tinuing to develop improvements to its Integrated 
Property System. At the time of our follow-up, it 
was developing and testing some system changes 
and working to add some automatic control func- 
tions to improve the efficiency and accuracy of 
reviews. It planned to implement these changes by 
the end of 2012. 

EXPENDITURES 

Establishing the Need for Goods and 
Services, Acquisition Process for Goods 
and Services 

Recommendation 7 

To ensure that goods and services are acquired only 
when necessary and are the most appropriate in the 
circumstances, the Municipal Property Assessment 
Corporation (Corporation) should comply with its own 
procurement policy and ensure that each acquisition is: 

• justified based on clear business requirements; 

• the most appropriate option to satisfy the busi- 
ness requirement under the circumstances; and 

• supported by a properly authorized purchase 
requisition that provides evidence of the author- 
ization to proceed. 

To ensure that all vendors are treated fairly and 
equitably and that it obtains value for money spent, 
the Corporation should also: 

• acquire goods and services competitively in 
compliance with its own requirements and those 
of the Ministry of Finance; and 

• prepare and maintain, for each transaction, 
adequate documentation to demonstrate why 
the successful vendor was selected. 

Status 

In its response to our 2010 Annual Report, the 
Corporation indicated that it had implemented a 
new Procurement Policy in 2009 to conform to the 
province's procurement directive. The Corporation 
has further revised the policy, and received board 
approval of the changes in September 2011. The 
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new policy requires the justification for each acqui- 
sition to be based on clear and documented busi- 
ness requirements. It also requires the use of the 
Integrated Resources Information System (IRIS) for 
requisitions over $10,000. These purchases cannot 
be processed without proper electronic approval in 
accordance with a Delegation of Authority Frame- 
work built into the system. 

The new Procurement Policy also requires goods 
and services to be acquired competitively with a 
few exceptions. Goods and non-consulting services 
valued over $100,000 must be procured through a 
Request for Proposal (RFP) process, and all RFPs 
must be publicly posted. For acquisition of consult- 
ing services valued at $100,000 or more, an open 
competitive procurement process generally must 
be used. A minimum of three vendors are required 
to submit a written proposal for consulting servi- 
ces valued less than $100,000. The Corporation 
informed us that it would use the government's 
vendors of record where appropriate. 

The Corporation informed us that its Procure- 
ment Unit must maintain a full record of the pro- 
cess used to demonstrate and support the vendor 
selected for each RFP. For purchases that are over 
$10,000 but not tendered, the supporting docu- 
ments must be included with the purchase requisi- 
tion in the IRIS system. 

The Corporation advised us that its internal audit 
service unit was planning a compliance review of 
the Procurement Policy by the end of 2012. 

Contractual Agreements 

Recommendation 8 

The Municipal Property Assessment Corpora- 
tion should adhere to good business practices by 
ensuring that: 

• it enters into appropriate written agreements 
with all of its of suppliers of goods and services 
and that these written agreements include all 
the normally expected terms and conditions, 
such as ceiling prices, expected deliverables, and 
associated time frames; 



• all such agreements are approved by individuals 
with the authority to do so; 

• supplier invoices contain sufficient detail so that 
the reasonableness of amounts billed and paid 
can be assessed; and 

• it assesses and adequately documents the quali- 
fications and performance of suppliers of goods 
and services. 

Status 

The Corporation advised us that it had established 
written service agreements with all of its suppliers 
and contractors in the fall of 2009. These agree- 
ments include terms and conditions such as ceiling 
price, expected deliverables, associated time frames 
and the circumstances under which price increases 
would be permitted. 

As mentioned in the previous section, the Cor- 
poration's new Procurement Policy requires that 
purchase requisitions be recorded in IRIS for each 
supplier or contractor. The purchase requisition 
requires electronic approval consistent with the 
Delegation of Authority. Once approved, a purchase 
order and a service agreement are created. Accord- 
ing to the Corporation, all existing service agree- 
ments are signed by the accountable manager, as 
per the Delegation of Authority Framework. 

With regard to payments to contractors, the 
Corporation informed us that, effective Janu- 
ary 2010, accountable managers are required to 
document the time spent on each project by each 
contractor, and provide this information to the Pro- 
ject Management Office. The Project Management 
Office is then to match it to contractor invoices in 
the IRIS system, ensuring there is sufficient sup- 
port before forwarding the invoices to the account- 
ing department for payment. 

The Corporation also informed us that at the 
end of each contract, managers must also review 
contractor performance, and these evaluations are 
stored in the IRIS system. The Corporation had 
established a standard template to evaluate con- 
tractor performance, but was still in the process of 
developing a similar template to evaluate supplier 
performance at the time of our follow-up. 
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The Corporation's internal audit unit was 
planning a compliance review of the Procurement 
Policy, which is to include a review of its new ser- 
vice agreement practices, by the end of 2012. 

Travel, Meals and Hospitality 

Recommendation 9 

The Municipal Property Assessment Corporation 
(Corporation) should consult with the Ministry of 
Finance to determine whether it is the Ministry's 
intention to have the Corporation comply with the 
spirit and intent of the government's own directive 
for the reimbursement of travel, meal, and hospital- 
ity expenses. As well, the Corporation needs to adopt 
more rigour in enforcing its travel, meal, and hospi- 
tality policies. 



Status 

The Corporation advised us that it had fully 
implemented the government's Travel, Meal and 
Hospitality Expenses Directive as of January 2011 
and provided the directive to all managers to make 
sure they were aware of the changes and new 
requirements. 

The Corporation informed us that, in January 
2010, it added control checks to the IRIS system to 
flag possible duplicate expenses for investigation. 
In October and November 2011, the Corporation's 
Finance Branch reviewed about 31% of the total 
expenses claimed for these two periods and found 
that seven meal claims, worth about $110, were in 
fact duplicated claims. The claims were corrected 
prior to expense reimbursement. The branch did 
not identify any subsequent non-compliance and 
informed us that it planned to continue with com- 
pliance reviews on at least 5% of the total expenses 
claimed on a weekly basis. 
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Non-hazardous waste includes non-recyclable and 
recyclable materials generated by households and 
by businesses and organizations in the industrial, 
commercial and institutional (IC&I) sector. At the 
time of our 2010 audit, approximately 12.5 million 
tonnes of non-hazardous waste were being gener- 
ated in Ontario annually. The IC&I sector generated 
about 60% of this waste, and households — the 
residential sector — generated 40%. Non-hazardous 
waste is managed in two main ways: by disposal 
(usually in a landfill or incineration) or by diversion 
(for example, recycling) . About two-thirds of the 
province's disposed waste is deposited in landfills 
in Ontario, with the rest shipped to landfills in the 
United States. 

Municipal governments are generally respon- 
sible for managing waste generated by the resi- 
dential sector. The IC&I sector and most multi-unit 
residential buildings are responsible for managing 
the waste they produce and typically use private- 
sector companies to transport the waste to landfills 
or recycling facilities. 

The Ontario government, primarily through 
the Ministry of the Environment (Ministry), is 
responsible for setting standards for the manage- 
ment of non-hazardous waste through legislation 



and regulations and for enforcing compliance with 
these legislative requirements. The Ministry is also 
responsible for approving new municipal and pri- 
vate-sector waste management sites and facilities 
and for ensuring that these operations comply with 
legislative requirements. In Ontario, the manage- 
ment of non-hazardous waste is governed primarily 
by the Environmental Protection Act (EPA) , the 
Environmental Assessment Act (EAA), and the Waste 
Diversion Act, 2002 (WDA). 

Under the WDA, the provincial government has 
established an arm's-length organization, governed 
by a board of directors, called Waste Diversion 
Ontario (WDO). The key responsibility of WDO 
is to develop, implement and operate diversion 
programs for certain wastes, as designated by the 
Minister of the Environment, and to monitor the 
effectiveness and efficiency of those programs. It 
does this in conjunction with an Industry Fund- 
ing Organization (IFO) consisting of industry 
"stewards" — brand owners and first importers of 
products that generate the waste. At the time of our 
2010 audit, three IFOs had also been established: 
Stewardship Ontario, for blue box waste and muni- 
cipal hazardous or special waste; Ontario Electronic 
Stewardship, for waste electrical and electronic 
equipment; and Ontario Tire Stewardship, for 
used tires. 

In 2004, the government set a goal of diverting 
60% of Ontario's waste from being disposed in 



landfills by the end of 2008. At the time of our 2010 
audit, the combined diversion rate of waste gener- 
ated by the residential and IC&I sectors was about 
24%. This diversion rate ranked Ontario as sixth 
among the provinces. Waste diversion in the resi- 
dential sector, at about 40%, had increased fairly 
substantially since 2002, but this increase had been 
offset by a drop in the IC&I sector's diversion rate. 

We noted that many of the issues that the 
government identified in 2004 as keys to achieving 
60% waste diversion by the end of 2008 had yet to 
be successfully addressed. Our specific observations 
were as follows: 

• Municipalities (generally responsible for 
managing residential waste) and households 
were making progress in diverting waste away 
from landfills. However, although their overall 
diversion rate for residential waste was about 
40%, we found that individual municipalities' 
diversion rates reported to us varied signifi- 
cantly, from about 20% to more than 60%. 
This was mainly due to differences in the fre- 
quency and quantity of disposable waste col- 
lection, differences in the blue box recyclable • 
materials that were collected and the fact that 
only some municipalities had implemented 
organic-waste composting programs. Other 
differences in municipalities' waste manage- 
ment practices included the following: 

• Whether a municipality can market its blue 
box and organic recyclable waste. Munici- 
palities compete with each other and with 
the private sector for markets for recyclable 
waste. The larger municipalities, which can 
generate significant volumes, are more suc- 
cessful at securing markets than the smaller 
municipalities and therefore can encourage 
greater recycling. 

• Cost. On average, municipalities reported 
that the cost of diverting a tonne of blue 
box recyclable materials was about 40% 
higher than the cost of disposing of a tonne 
of waste in a landfill. More than half of 
the municipalities that responded to our 
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survey indicated that the funding they 
received under the cost-sharing formula 
from industry "stewards" to offset some of 
the costs they incurred for running the blue 
box program was not sufficient. 

• Landfill capacity that is available to a muni- 
cipality. When waste is collected less often 
and bag limits are imposed, residents typ- 
ically divert more waste. For example, one 
municipality indicated that by collecting 
recyclable materials weekly and disposable 
waste every two weeks while imposing a 
bag limit, it was able to increase its diver- 
sion rate by about 20%. But the responses 
to our survey indicated that municipalities 
that have sufficient landfill capacity are less 
likely to limit the frequency of waste collec- 
tion and impose a bag limit on residents. 

• Residents' preferences. Municipal councils 
are well aware that their constituents want 
a higher level of waste pickup service and 
no bag limits regardless of the impact on 
waste diversion. 

The IC&I sector generated approximately 60% 
of the waste in Ontario, but managed to divert 
only about 12% of its waste. Regulations 
under the Environmental Protection Act require 
large generators to conduct a Waste Audit, 
prepare a Waste Reduction Work Plan, and 
implement programs to source-separate waste 
for reuse or recycling. However, the Ministry 
had little assurance that the regulations were 
being complied with for the following reasons: 

• The Ministry did not have adequate infor- 
mation on either the number of businesses 
or organizations to which the regulation 
applied or which segments of the IC&I sec- 
tor generated the largest amounts of waste 
so that it might target them for inspection. 

• In half of the inspection files we reviewed, 
there was no evidence that the ministry 
inspector had reviewed either the Waste 
Audit or the Waste Reduction Work Plan. 
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• The inspections did not assess the extent 
to which the IC&I-sector businesses were 
separating recyclable waste or whether the 
waste that had been source-separated was 
actually being processed for recycling. 

• Organic waste generated by both the residen- 
tial and IC&I sectors represented almost one- 
third of the total waste generated in Ontario, 
but there was no province-wide organic waste 
diversion program or target, despite the 
Ministry's having considered establishing a 
program as early as 2002. 

• One in five municipalities that responded 
to our survey felt that they had insufficient 
landfill disposal capacity for their residential 
waste. As well, the existing capacity was 
expected to be filled more quickly once export 
of residential waste to Michigan largely ended 
after 2010 and an additional 1 million tonnes 
of this waste previously shipped to that state 
began being deposited in Ontario landfills 
annually. Opening new landfills within muni- 
cipalities is not always a viable option, both 
because they are costly and because residents 
do not support new landfills. 

• The Ministry inspected landfills and non- 
hazardous waste management sites, facilities 
and systems against the conditions of their 
certificates of approval. But we noted that 
many of these certificates did not reflect 
changes in standards. Also, in our review 

of inspection files, we found that numerous 
examples of non-compliance with the certifi- 
cates' conditions had been noted, but many of 
these were not being followed up on a timely 
basis to ensure that the required changes 
were made. 
We made a number of recommendations for 
improvement and received commitments from 
the Ministry that it would take action to address 
our concerns. 



STANDING COMMITTEE ON 
PUBLIC ACCOUNTS 

The Standing Committee on Public Accounts held 
a hearing on this audit in February 2011. In May 
2011, the Committee tabled a report in the Legis- 
lature resulting from this hearing. The report con- 
tained nine recommendations, requesting that the 
Ministry report back to the Committee with respect 
to the following: 

• when the Ministry estimates it will reach its 
goal of 60% diversion of Ontario's waste from 
landfills and whether any additional tools are 
required for this; 

• the Ministry's strategy for increasing the rate 
of waste diversion in the IC&I sector, includ- 
ing a diversion target and timeline for the 
sector, ways that the Ministry will motivate 
businesses and organizations that are not 
regulated under the Environmental Protection 
Act to improve their diversion rates, and how 
the Ministry will monitor the businesses 
and organizations; 

• the Ministry's plans for obtaining adequate 
information on the number of businesses and 
organizations to which the waste diversion 
regulations apply and on which of these are 
the largest waste generators; 

• the Ministry's 2011 protocols for field inspect- 
ors, including whether inspections are risk- 
based and target the largest waste generators, 
how the Ministry tracks whether businesses 
and organizations have implemented their 
Waste Audits and Waste Reduction Work 
Plans, and how the Ministry determines 
whether businesses and organizations are 
source-separating waste and whether the 
source-separated waste is being recycled; 

• how the Ministry's new guidelines will 
increase the diversion of organic waste; 

• the Ministry's public consultations on and 
review of the Waste Diversion Act, 2002; 

• how the Ministry will improve its oversight of 
Waste Diversion Ontario (WDO) and whether 
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it will consider having a senior ministry repre- 
sentative sit on WDO's board of directors; and 
• the average time the Ministry takes to approve 
a certificate-of-approval application under 
its new approvals system and whether it has 
a standard for how long it takes to review 
an application. 
The Committee also requested that WDO 
report back on how it monitors whether its diver- 
sion programs are meeting targets, what steps it 
takes to address setbacks in meeting targets, and 
how it assesses the waste-diversion information it 
receives from municipalities and Industry Funding 
Organizations. 

The Ministry formally responded to the Com- 
mittee in October 2012. A number of issues raised 
by the Committee were similar to our observations. 
Where the Committee's recommendations were 
similar to ours, this follow-up includes the recent 
actions reported by the Ministry to address the 
concerns raised by both the Committee and our 
2010 audit. 



Status of Action Taken 
on Recommendations 



According to information provided to us by the 
Ministry, some progress has been made in address- 
ing many of the recommendations we made in our 
2010 Annual Report. For example, governance at 
Waste Diversion Ontario has been strengthened 
and Ontario's compost framework has been 
updated to encourage more composting. However, 
several recommendations will require more time to 
be addressed fully. In particular, our concerns with 
regard to diversion of waste in the IC&I sector have 
yet to be substantially addressed. In this regard, 
we note that according to Statistics Canada, the 
waste diversion rate in the IC&I sector in 2008 (the 
latest year for which information was available) 
was 12.7%, up only slightly from 12% in 2006. 



This and recent ministry inspection results suggest 
that much remains to be done to increase the waste 
diversion rate in the IC&I sector. The status of 
actions taken on each of our recommendations at 
the time of our follow-up was as follows. 

WASTE DIVERSION 

Residential-sector Waste 

Recommendation 1 

To further increase diversion of waste in the resi- 
dential sector, and as part of its current review of 
the Waste Diversion Act, 2002, the Ministry of the 
Environment should work with municipalities, indus- 
try "stewards, " and other stakeholders to: 

• increase the availability of reliable and sustain- 
able markets for recyclable and organic waste; 

• increase capacity within the province to process 
recyclable materials and organic waste; and 

• review the current funding formula for the blue 
box program to ensure that it achieves its object- 
ive of municipalities and "stewards" equally 
sharing costs. 

Status 

At the time of our follow-up, the Ministry indicated 
that in 2011, Stewardship Ontario had undertaken 
a market development program that identified new 
opportunities to improve the recycling chain and 
extract more value from recyclable materials in 
Ontario. Specifically, the Ministry informed us of 
the following: 

• In spring 2011, Stewardship Ontario issued 
a Request for Expressions of Interest (RFEI) 
aimed at identifying new companies with 
innovative approaches to recycling. More than 
60 submissions were received from companies 
interested in forming commercial relation- 
ships with Stewardship Ontario. As part of 
this initiative, Stewardship Ontario invested 
$500,000 in a company with a view toward 
commercializing new plastics recycling 
technology and, according to the Ministry, 
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was in active discussions with more than 15 
other companies. 

• Stewardship Ontario provided a loan to an 
existing plastics recycling company to enable 
it to relocate and further expand its facility. 

• Stewardship Ontario initiated a review of 
recycling of fibre packaging and the poten- 
tial to expand collection and recycling of 
these materials. 

As prescribed in the WDA, the blue box funding 
formula requires industry to fund 50% of the net 
cost of the municipal blue box program. As noted 
earlier, more than half of the municipalities that 
responded to the survey we conducted as part of 
our 2010 audit indicated that the funding they 
received under the cost-sharing formula to offset 
some of the cost they incurred for running the blue 
box program was not sufficient. At the time of our 
follow-up, the Ministry indicated to us that any 
changes to the funding allocation model are pro- 
posed through WDO's Municipal Industry Program 
Committee, on which municipalities and Steward- 
ship Ontario are equally represented, and that a 
review of the funding formula was under way. 

Industrial, Commercial, and Institutional 
(IC&I) Sector Waste 

Recommendation 2 

In order to increase waste diversion in the IC&I sector, 
the Ministry of the Environment should: 

• gather information on the amount and type of 
waste generated by small and medium-sized 
businesses and organizations that are not regu- 
lated under the Environmental Protection Act 
(EPA) and consider what actions could be taken 
to reduce the amount of waste that is currently 
going to landfills; 

• require those large entities that are regulated 
under the EPA to publicly report their waste 
diversion rates. The Ministry should then, as 
part of its inspection work, assess the accuracy 
of the rates reported; and 



• conduct research into successful practices used 
in other provinces and European countries 

to divert IC&I-sector waste from landfilb. In 
assessing which practices might be transferable 
to Ontario, the Ministry will need to balance 
the environmental benefits with the economic 
challenges currently being faced by the 
business community. 

Status 

At the time of our follow-up, the Ministry indicated 
that it was continuing to consider ways of obtaining 
appropriate information necessary to support 
diversion policies and programs in this sector, and 
conducting further jurisdictional research as part of 
its ongoing policy development work. 

The Ministry had also supported the October 
2011 launch of the 3RCertified Waste Diversion 
Certification Program by the Recycling Council 
of Ontario (RCO), a not-for-profit organization 
involved in policy, education and work surround- 
ing waste generation and diversion. This program 
gives businesses and institutions the opportunity to 
become accredited with the 3RCertified standard 
by demonstrating conformance to a set of criteria, 
verified by an onsite evaluation by the RCO. Com- 
pliance with the Ministry's regulations for the man- 
agement of non-hazardous waste is also one of the 
requirements to achieve base certification under 
the program. 

At the time our report was being finalized, the 
RCO announced that four organizations in the 
province had achieved 3RCertification. 

Compliance in the IC&I Sector, Scope 
of Inspections in the IC&I Sector and 
Enforcement of Other EPA Regulations 

Recommendation 3 

To improve waste diversion in the IC&I sector, the 
Ministry of the Environment should: 

• gather data on the number of businesses to which 
the waste diversion regulations apply and on 
which of these are the largest waste generators 



Non-hazardous Waste Disposal and Diversion 



to assist both its inspection activities and policy 
decisions, and ensure that businesses are aware 
of the requirements of the regulations; 

• increase the scope of its inspections to include 
an assessment of the extent to which businesses 
have implemented their Waste Audits and 
Waste Reduction Work Plans and whether there 
has been any increase in the amount of waste 
diverted; and 

• verify during inspections and document whether 
waste management companies are operating 
under a valid certificate of approval. 

If the Ministry plans to continue not to enforce 
its regulation that requires large manufacturers, 
packagers, and importers to implement a packaging 
reduction plan and its regulation that requires all car- 
bonated soft drinks to be sold in refillable containers, 
it should consider revoking these regulations. 

Status 

At the time of our follow-up, the Ministry indicated 
that, as an overall measure aimed at increasing 
waste diversion, it was continuing to work with 
all its partners, including businesses and organ- 
izations and the Canadian Council of Ministers 
of the Environment, to reduce the amount of 
packaging created. 

In our 2010 Annual Report, we noted that the 
Ministry did not have adequate information on the 
number of businesses and organizations to which 
the EPA's waste diversion regulations applied. This 
observation prompted us to recommend to the Min- 
istry to gather such information and to ensure that 
businesses and organizations are aware of the regu- 
latory requirements. At the time of our follow-up 
the Ministry provided us with the sources of infor- 
mation it uses to identify businesses and organiza- 
tions covered by the EPA regulations. The Ministry 
indicated that, based on these sources, it now had 
an estimate of the number of facilities to which the 
waste diversion regulations apply. The Ministry 
also informed us that over the last two years it 
had begun working with 14 businesses (with a 
combined total of more than 550 retail locations) to 
develop company-wide waste diversion programs. 



The Ministry also indicated that, in the last two 
years, it had conducted outreach efforts tailored to 
individual sub-sectors. For instance, the Ministry 
had worked with 11 school boards and with the 
Ontario Hospital Association to customize Waste 
Audit and Waste Reduction Work Plan guides to 
help the multiple facilities in each sub-sector meet 
their regulatory requirements. 

In the 2011/12 fiscal year, the Ministry 
conducted a "re-sweep" of a select number of 
previously inspected facilities to assess the impact 
of inspection efforts on, for example, the extent to 
which the facilities were preparing Waste Audits 
and Waste Reduction Plans and source-separating 
required wastes. Out of 104 selected sites, 17 had 
changed operations or ownership or had closed. Of 
the remaining 87, only 21 passed the re-inspection. 
The Ministry informed us that as of August 2012, 
65 of the 66 sites that had failed the re-inspection 
had been brought into compliance. There was an 
Order issued against the remaining site with a 
compliance date of August 31, 2012. The Ministry 
also indicated that based on the results of the 
re-inspections, recommendations for program 
improvements would be made. 

A number of the inspection files of waste 
management companies that we reviewed as part 
of our 2010 audit did not contain evidence that 
the inspector had checked that the company was 
operating under a valid Ministry-issued certificate 
of approval. The Ministry informed us that chan- 
ges implemented to its inspection tracking and 
reporting system now ensure that whether waste 
management companies are operating under a 
valid certificate of approval is consistently docu- 
mented within the system. 

Lastly, the Ministry indicated to us that, as part 
of its initiative to reduce unnecessary or obsolete 
regulatory requirements, it was still considering 
whether the regulation that requires large manu- 
facturers, packagers and importers to implement a 
packaging reduction plan and the regulation that 
requires all carbonated soft drinks to be sold in 
refillable containers should be revoked. 
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Organic Waste 

Recommendation 4 

To increase overall waste diversion in Ontario, the 
Ministry of the Environment should work with 
municipalities, businesses and organizations, and 
private-sector waste management companies to phase 
in over time a province-wide organic waste diversion 
program for both the residential and IC&I sectors. 
As part of implementing the program, the Ministry, 
in conjunction with these stakeholders, will need to 
ensure that there is sufficient capacity to process the 
additional organic waste and that a sustainable mar- 
ket exists for the processed waste. 

Status 

The Ministry indicated that it had completed 
consultations with the public and industry on a 
proposed update to Ontario's compost framework, 
including appropriate standards for compost, 
environmental protection measures and other 
tools to support a sustainable market and process- 
ing capacity for organic waste. At the time of our 
follow-up, the Ministry had released an update to 
the framework that included changes to compost 
quality categories and standards, and best practices 
for the siting, design, operation and maintenance 
of composting facilities. These changes were to take 
effect January 1, 2013. 

Waste Diversion Ontario 

Recommendation 5 

To enhance accountability for the achievement of 
diversion targets for wastes specifically designated 
under the Waste Diversion Act, 2002, and to ensure 
that the reporting of the diversion results against the 
targets to the Minister is complete and reasonably 
accurate, the Ministry of the Environment should: 
• review the operating agreement to ensure that it 
contains sufficient accountability provisions to 
require Waste Diversion Ontario to provide an 
action plan when waste diversion targets are not 
being met; 



• ensure that the waste diversion information 
submitted by municipalities and the Industry 
Funding Organizations (IFOs) is objectively 
assessed, including the impact on this informa- 
tion of unregistered collectors that do not submit 
waste diversion data; and 

• reconsider its policy of allowing collectors of 
designated wastes the option of whether or not 
to register with an IFO. 

Where retailers are charging a specific "ecofee," 
the Ministry should also reconsider whether they 
should be required to disclose the amount of the fee on 
the customer receipt. 

Status 

On October 25, 2010, after the completion of 
our 2010 audit, the Minister of the Environment 
advised WDO of the need to amend its board struc- 
ture so that it: 

• reflected the knowledge and expertise 
required to oversee waste diversion programs 
under the Waste Diversion Act, 2002 (WDA); 

• avoided real, potential or apparent conflicts 
of interest between WDO board members and 
the programs they oversee; and 

• included at least one board member selected 
from a consumer-focused organization to 
ensure that a consumer perspective would be 
brought to the WDO board. 

In a letter dated February 9, 2012, the Minister 
stated that he had accepted in principle the pro- 
posal submitted by WDO's board on May 27, 2011, 
with regard to a new governance structure. WDO 
transitioned to the new board in April 2012. 

In the February 9 letter, the Minister also 
directed WDO to undertake a detailed review of all 
diversion program budgets and to regularly monitor 
expenditures to ensure that there are realistic and 
cost-effective plans to achieve IFO performance 
targets. In this regard, WDO is required to report 
quarterly to the Ministry. The Minister also required 
WDO to implement an effective monitoring pro- 
gram to review IFOs' annual program performance 
and the achievement of plan targets, to promptly 
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take corrective action on performance issues and to 
report quarterly on these matters to the Ministry. 
At the time of our follow-up, the Ministry informed 
us that, in addition to the existing requirement for 
audited financial statements, WDO had begun to 
implement independent third-party verification 
of environmental performance and standard- 
ized reporting for all waste diversion programs. 
However, the Ministry also informed us that since 
the WDA doesn't compel individuals or companies 
who collect waste to share information on their 
activities, the third-party verification is limited to 
those companies that participate in the diversion 
programs. The audits of the programs for blue box 
and municipal hazardous or special waste and for 
waste electrical and electronic equipment had been 
completed and were being reviewed by WDO before 
being made public. The audit of the used-tires 
program had been deferred by WDO pending an 
evaluation of the performance audits undertaken 
for the other waste diversion programs. 

At the time of our follow-up, no action had been 
taken on our recommendation to the Ministry to 
reconsider its practice of allowing collectors of 
designated wastes the option of whether or not to 
register with an IFO. The Ministry informed us that 
it would consider this recommendation as it works 
toward making the province's waste diversion 
framework more effective. 

The Ministry also informed us at the time of our 
follow-up that, to investigate whether retailers are 
charging incorrect or misleading eco fees, environ- 
mental compliance officers have been conducting 
site visits since November 2010 as a follow-up to 
consumer hotline calls and undertaking mystery 
shopping activities. The Ministry indicated that it 
had conducted 1,303 mystery shopping activities 
as of early January 2012 and that 252 (19%) of the 
sites involved had been referred to the Ministry's 
Investigation and Enforcement Branch to assess 
whether there was evidence of a violation that 
needed to be referred to the Ministry of Consumer 
Services for prosecution under the Consumer Pro- 
tection Act, 2002. 



WASTE DISPOSAL 

Recommendation 6 

To increase Ontario's capacity to dispose waste, the 
Ministry of the Environment should take a leader- 
ship role in working with municipalities and other 
stakeholders to research and adopt alternative waste 
disposal technologies such as the thermal treatment 
facilities that are in use in other jurisdictions. 

Status 

At the time of our follow-up, the Ministry indicated 
that the province's priority was still to divert as 
much waste as possible from disposal through the 
3Rs (reduce, reuse and recycle). Generating energy 
from waste is an option that can be considered to 
help manage the residual waste that remains after 
diversion. In this regard, the Ministry informed 
us that regulations that came into effect in 2007 
had streamlined the approvals process for pilot 
and demonstration energy-from-waste facilities. 
The Ministry indicated that it had approved under 
these regulations a number of pilot and demonstra- 
tion facilities incorporating energy-from-waste 
technologies. For example, in July 2011, a cer- 
tificate of approval was issued for the York and 
Durham Regions' energy-from-waste facility. This 
facility is to have the capacity to receive 140,000 
tonnes of residual waste annually and the ability 
to generate up to 20 megawatts of energy. Also, 
an approval was issued in October 2011 for the 
permanent operation of a demonstration energy- 
from-waste facility in Ottawa. 

Monitoring of Waste Disposal Sites and 
Waste Management Systems 

Certificates of Approval 
Recommendation 7 

To better facilitate compliance with certificates of 
approval for non-hazardous waste management 
sites, facilities, and systems, the Ministry of the 
Environment should: 

• review its existing certificates, especially for 
the larger or more environmentally risky 
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operations, to ensure that they reflect current 
standards and operations and revise those that 
need updating; 

• in cases where numerous amendments have 
been issued to an existing certificate, consolidate 
the amendments into one, new certificate; 

• develop a standard for the time it should take 
to review certificate-of-approval applications 
for non-hazardous waste operations and 
review the outdated application fee it charges to 
ensure that it reflects the cost of processing the 
applications; and 

• collect, follow up on, and review the re- 
valuation of the required financial assurance, 
especially for the larger operators, on a 
timely basis. 

Status 

At the time of our follow-up, the Ministry indicated 
that it had completed a review of the Environ- 
mental Compliance Approvals (ECAs, formerly 
certificates of approval) for 32 larger landfill sites 
that together received more than 85% of the waste 
destined for landfill in Ontario. The review, accord- 
ing to the Ministry, concluded that most approvals 
were current and that only one site required a 
minor update, which was later issued. 

Also, the Ministry informed us, as it did in 2010, 
that it had revised the practice of amending ECAs. 
A single ECA is now to be issued and is to include 
the original as well as any subsequent notices 
of amendment. 

The Ministry also informed us that its Environ- 
mental Approvals Access and Service Integration 
Branch had begun providing monthly reports to 
regional directors on outstanding financial assur- 
ance matters to ensure that field staff were follow- 
ing up with ECA holders. 

Over the next two years, the Ministry indicated 
that it planned to continue modernizing the 
approvals program in ways that would address 
our other recommendations aimed at developing a 
standard for the time it should take to review ECA 
applications for non-hazardous waste operations 



and reviewing the application fee to ensure that it 
reflects the cost of processing the applications. 

Inspections 

Recommendation 8 

To improve its monitoring of non-hazardous waste 
management operations for compliance with legislative 
requirements, the Ministry of the Environment should: 

• impose timeframes for corrective action where 
inspections detect cases of non-compliance, and 
follow up to ensure that the required remedial 
action has been taken within the required 
timelines; and 

• ensure that time-sensitive materials such as 
annual reports from non-hazardous waste man- 
agement operations are submitted and reviewed 
on a timely basis. 

Status 

At the time of our follow-up, the Ministry had 
implemented updated monitoring procedures, 
including a time frame for finalizing actions of 45 
to 60 days from sign-off of inspections that detect 
cases of non-compliance. The updated procedures 
were communicated to staff in February 2011. 
Training sessions for environmental officers on the 
updated monitoring procedures were held dur- 
ing spring 2011. The Ministry indicated that the 
training also covered the Ministry's data system 
functionalities that track and provide automated 
alerts on the status of abatement actions. According 
to the Ministry, through the combination of train- 
ing and better tracking of the status of abatement 
actions, it had improved the documentation and 
follow-up of remedial actions where inspections 
detect non-compliance. 

The Ministry had also developed a risk-based 
approach to address the submission and review of 
annual reports from waste disposal sites. Environ- 
mental officers were trained in March and April 
2011 on the new procedures for annual report 
assessment. According to the Ministry, the training 
provided to environmental officers, together with 
guidance material developed for the stakeholder 



Non-hazardous Waste Disposal and Diversion 



community to improve the quality of submissions, 
had led to a more effective review process. 

MEASURING PROGRESS IN 
WASTE DIVERSION 

Recommendation 9 

The Ministry of the Environment should assess the 
benefits of adopting an alternative performance 
indicator, such as the per capita waste disposal rate, 
because it is more straightforward to calculate and is 
likely a more accurate and reliable measure of waste 
diversion in Ontario that will facilitate benchmarking 
progress relative to other jurisdictions. 



Status 

At the time of our follow-up, the Ministry indicated 
that it was using a variety of waste diversion data 
sources to assist in evaluating waste diversion 
progress, including data from Waste Diversion 
Ontario, Statistics Canada, municipalities and 
waste management companies, as well as informa- 
tion from the Ministry of Finance on the Ontario 
Deposit Return Program. The Ministry informed 
us that it was continuing to assess the benefits of a 
range of performance indicators related to waste 
diversion, including the use of a per capita waste 
disposal rate. 




Chapter 4 
Section 
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Organ and Tissue Donation 
and Transplantation 

Follow-up to VFM Section 3.10, 2010 Annual Report 



Background 




In the 2011/12 fiscal year, almost 1,000 organs 
were transplanted from almost 550 donors at the 
eight Ontario hospitals that perform transplants. As 
of March 31, 2012, more than 1,500 Ontarians were 
waiting for organs, most of them for a kidney or a 
liver. As well as saving or enhancing lives, trans- 
plants can save money. For example, each kidney 
transplant surgery cost about $25,000 at the time 
of our 2010 audit. The same year, however, the cost 
of dialysis — a mechanical procedure carried out 
frequently to cleanse the blood of a person whose 
kidneys have failed — was about $70,000 a year. 

The Trillium Gift of Life Network (Network), 
which has a staff of about 130, was established 
in 2002 as an agency of the Ministry of Health 
and Long-Term Care (Ministry) to co-ordinate the 
donation of organs and tissue, which includes eyes 
and bones. Funding to the Network and the eight 
hospitals for transplants in the 2009/10 fiscal year 
was about $100 million. Our 2010 audit assessed 
whether there were adequate policies, procedures 
and systems in place to meet the organ and tissue 
needs of Ontarians in an efficient and fair manner. 

As we reported in our 2010 Annual Report, 
initiatives by the Network, the Ministry and the 
transplant hospitals had improved the province's 



ability to meet organ- and tissue-transplant needs. 
As demand exceeds availability for many organs, 
the willingness of families of deceased people 
to donate organs is critical. Since the Network's 
establishment, the number of deceased donors 
has increased from 11.3 to 16.7 donors per million 
people. Nevertheless, our 2010 audit suggested 
certain changes that could be made to reduce wait 
times for organs, thus saving lives and improving 
patients' quality of life. 

Our findings included the following: 

• There was no periodic independent review of 
the Network's allocation of organs. Oversight 
of transplant activities on a province-wide 
basis needed to be enhanced to help ensure 
that patients were consistently prioritized on 
wait lists and that the highest-priority patient 
received the first available compatible organ. 
In more than 40% of the cases we reviewed, 
the highest-priority patient did not get the 
organ, and no reason for this decision was 
documented. 

• Forty hospitals generally did not refer poten- 
tial donors to the Network even though they 
had the medical equipment necessary to 
maintain organs until transplant. 

• For years, many Ontarians signed the 
donation-consent part of their driver's 
licence and kept it in their wallet. However, 
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this type of consent was not included in the 
Ministry's consent registry, which the Network 
uses to determine if a potential donor has 
given consent. 

There was a lack of consistent clinical criteria 
regarding the time when hospitals should 
refer potential donors to the Network, 
resulting in many referrals being made too 
late or not at all. 

Only 15,000 of the 4 million Ontarians who 
still have red-and-white OHIP cards had their 
consent registered with the Ministry (almost 
certainly because this required mailing in 
a separate form to ServiceOntario), while 
1.9 million people with photo OHIP cards had 
registered (because people are specifically 
asked during the application/renewal process 
if they want to register) . Consent-registration 
rates also varied significantly among regions, 
from less than 10% in Toronto to more than 
40% in Sudbury. 

Hospitals indicated that patients requiring 
organs were not always referred so that the 
transplant hospital could determine whether 
the patient was eligible to be placed on a 
transplant wait list. For example, only 13% 
of dialysis patients were on a kidney wait list, 
and rates varied from only 3% in the South 
East Local Health Integration Network (LHIN) 
to 16% in the Champlain LHIN. 
Instead of being allocated to the highest- 
priority patient province-wide, kidneys and 
livers generally stayed in the same region 
in which they were donated. As a result, for 
example, 90% of kidney recipients received 
the organ within four years in one area of the 
province, while those in two other regions 
waited about nine years. Wait times by organ 
type were generally not publicly available. 
Transplant hospitals did not have electronic 
access to donor information such as medical 
history and laboratory results, which is neces- 
sary to determine organ viability for potential 
recipients. Instead, they generally relied on 



the Network to verbally communicate this 
information, increasing the risk of decisions 
being based on incomplete or inaccurate 
information. 

• Less than 8% of tissue needed for transplant 
in Ontario actually came from Ontario donors 
because of a lack of resources to recover, 
process and store it. Instead, Ontario hospitals 
purchased tissue primarily from the United 
States and Quebec. 

• One Ontario hospital performed only six 
transplants in a year, and although Ontario 
does not stipulate a minimum yearly number 
of transplants to ensure a hospital remains 
proficient, the U.S. minimum requirement is 
generally 10. 

We made a number of recommendations for 
improvement and received commitments from the 
Network and the Ministry that they would take 
action to address our concerns. 



Status of Actions Taken 
Recommendations 



The Network and the Ministry provided us with 
information in the spring and summer of 2012 on 
the current status of our recommendations. Accord- 
ing to this information, some progress was made 
in implementing all of the recommendations in our 
2010 Annual Report, with significant progress in 
areas such as simplifying the consent registration 
process and using kidney pumps to increase the 
viability of kidneys from deceased donors. While 
more hospitals are now required to report potential 
organ and tissue donors to the Network, it will take 
additional time to fully address and implement this 
and several of our other recommendations. Some 
actions depend on the interprovincial-territorial 
response to the national organ and tissue donation 
and transplantation plan released in April 2011 by 
Canadian Blood Services. 
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The current status of the actions taken by the 
Network and the Ministry is summarized following 
each recommendation. 

ORGANS 

Identifying and Referring Donors 

Recommendation 1 

To increase the number of organs available to individ- 
uals waiting for a transplant, the Trillium Gift of Life 
Network (Network) could enhance the identification 
of potential organ donors through such means as: 

• determining whether all 61 hospitals with 
advanced ventilator capacity (necessary to 
maintain the viability of organs for transplant), 
rather than just the current 21 hospitals, should 
be required to notify the Network of potential 
organ donors, in accordance with the recom- 
mendation of the Ministry of Health and Long- 
Term Care's Organ and Tissue Transplantation 
Wait Times Expert Panel; 

• developing and implementing consistent, appro- 
priate clinical criteria, in conjunction with 
hospitals, to assist physicians in knowing when 
to notify the Network of potential donors; 

• using existing provincial systems, such as Criti- 
Call, a referral service for critically ill patients, 
and the Emergency Neurosurgery Image Transfer 
System, used to remotely view the computed 
tomography (CT) images that can confirm brain 
death, to help identify potential donors; and 

• working with all stakeholders — including the 
Ministry, hospitals, and physicians — to ensure 
that there are sufficient financial incentives to 
encourage more widespread identification and 
reporting of potential donors. 

Status 

At the time of our follow-up, the Network said that 
the requirement to report potential organ donors 
was being expanded to all hospitals that have 



advanced ventilator capacity. As of summer 2012, 
18 more hospitals were notifying the Network 
of potential organ donors, and another 10 were 
expected to start by the end of the 2012/13 fiscal 
year. The remaining hospitals with ventilator cap- 
acity were expected to begin notifying the Network 
in the 2013/14 fiscal year. 

The Network indicated that updated clinical 
triggers, which assist physicians in identifying when 
to notify the Network of potential donors, were 
reviewed and endorsed by the Network's Donation 
Steering Committee in fall 2011. The Network 
noted that the revised triggers were being incor- 
porated into the policies of all hospitals required 
to report potential organ donors to the Network. 
The Network also planned broader consultations 
to ensure acceptance of the criteria, and indicated 
that it may further revise the triggers after the con- 
sultations end in December 2012. 

The Network said it planned to review oppor- 
tunities to integrate organ donation and end-of-life 
care, including ways to leverage the Emergency 
Neurosurgery Image Transfer System and CritiCall, 
during the 2012/13 fiscal year. The Network noted 
that the Executive Director of CritiCall was a mem- 
ber of the Network's Donation Steering Committee. 

The Network said it worked with the Ontario 
Medical Association and physicians, in consultation 
with the Ministry, to identify gaps and oppor- 
tunities in the current physician fee schedule to 
encourage doctors to identify and report potential 
organ donors. The Ministry determined that the 
physician fee schedule already includes fees cover- 
ing this area — but that physicians may not be aware 
that it does because they seldom bill for it. The 
Ministry expected the Network to clarify with phys- 
icians when these fees can be billed and further 
indicated that it will monitor the implementation 
of the Health System Funding Strategy for hospital 
funding opportunities related to the identification 
and reporting of potential donors. 
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CONSENT 

Recommendation 2 

To help improve consent rates for potential organ 
donation, the Trillium Gift of Life Network (Net- 
work) should: 

• work with the Ministry of Health and Long- 
Term Care, the Ministry of Transportation, 
and ServiceOntario to change the system of 
obtaining consent at the time of driver's licence 
renewal to enable persons to be added to the 
donor registry, because neither the Network 
nor hospitals have access to the donor card 
previously sent with licence renewals that many 
people sign and keep in their wallet; 

• determine, in conjunction with the hospitals, 
the best approaches to increasing consent rates 
at the hospitals, especially in those areas of 
the province where consent rates are low — for 
example, by identifying specific individuals who 
have an aptitude for or training in successfully 
requesting consent; and 

• consider implementing a "mandatory ask" 
policy, along the lines of a policy used in the 
United States, which would require that the next 
of kin of every potential organ donor be asked 
for consent before the removal of life support. 

Further, the Ministry of Health and Long-Term 
Care should simplify the process by which people 
register consent to be an organ donor, such as by 
implementing an on-line consent registry similar 
to those available in British Columbia and other 
jurisdictions. 

Status 

In fall 2010, the Network, in collaboration with 
ServiceOntario and the ministries of Transportation 
and Health and Long-Term Care, began mailing 
donor-registration forms with prepaid return envel- 
opes to persons renewing their driver's licence. 
The Network indicated that the response rate has 
been much higher, with 4.3% of people signing 
their consent and returning the forms to Service- 
Ontario between December 2010 and March 2012. 
The Ministry further noted that the Network and 



ServiceOntario launched a pilot project in which 
ServiceOntario counter staff ask people who renew 
a driver's licence or obtain an Ontario photo health 
card if they wish to register as potential donors. 
The Network and ServiceOntario were planning to 
expand this practice province-wide by March 2013. 

In May 2011, the Network began to monitor 
the effectiveness of all staff involved in obtaining 
consent from families of potential donors, and 
found that staff with Network training and experi- 
ence were more successful at obtaining consent 
than health-care professionals with no training in 
this area. The Network's policy endorses its staff 
approaching families for consent, but also says 
that a discussion is required between Network staff 
and the health-care team of a potential donor to 
determine who should approach the family, and 
how best to do it. The Network provides quarterly 
training sessions for staff, and monitors all staff 
involved in obtaining consent to identify areas 
where additional follow-up training is necessary. 
The Network also noted that it held a workshop in 
March 2012 to help ensure staff have the appropri- 
ate knowledge and understanding to deal with 
families and health-care professionals from diverse 
cultural backgrounds. 

To improve consent rates, the Network's Dona- 
tion Steering Committee endorsed a "mandatory 
ask" policy in summer 2012 whereby consent to 
donation must be requested of all potential donors 
before the withdrawal of life-sustaining therapy. 
The Network also indicated that it was reviewing 
situations where organ donations do not proceed 
because a potential donor's family does not support 
the person's previously registered consent deci- 
sion, to see if it can devise strategies for ensuring 
that decisions are fully informed while the family's 
rights are still respected. 

The Ministry said an online donor registration 
system was implemented in spring 2011, and that 
more than 44,000 people used it to register their 
consent between June 2011 and April 2012. Also, 
the Ministry, working with the Ministry of Govern- 
ment Services and ServiceOntario, made online 
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registration accessible through mobile telephone 
and computing devices in April 2012. As well, the 
Network is working on an electronic "grass roots" 
social media campaign, whereby interested individ- 
uals can encourage others to register their consent 
to be organ and tissue donors. 

ORGAN WAIT-LISTS 

Recommendation 3 

To enhance its management of the wait-lists for organ 
transplants, the Trillium Gift of Life Network (Net- 
work), in conjunction with transplant hospitals and 
physicians, should: 

• develop target time frames for provincial 
priority rating scales for organ transplants, as 
recommended by the Ministry of Health and 
Long-Term Care's Organ and Tissue Trans- 
plantation Wait Times Expert Panel; 

• determine the best way to communicate referral 
criteria to non-transplant physicians, so that 
individuals who would benefit from a transplant 
(including from a quality-of-life perspective) are 
added to the wait-list; and 

• require hospitals to enter on the Network's 
system the reason for taking a patient off the 
wait-list, and periodically review, by hospital, 
the number of patients removed from the 
wait-list because they die or become too ill for a 
transplant, to determine whether actions can be 
taken to minimize the incidence of such cases. 

Status 

At the time of our follow-up, the Network said it 
had initiated a review of national and global prac- 
tices for establishing target wait times for organ 
transplants. In conjunction with clinical experts, 
the Network plans to use this information to help 
determine by fall 2013 the appropriate use of 
wait-time information for transplant patients. The 
Network noted that the development of target wait 
times also requires standardized organ-transplant 
listing criteria and organ-allocation practices. 



The Network indicated that its organ-specific 
working groups identified consistent practices for 
adding patients to transplant wait lists, and are 
working on standardized referral criteria. Once 
referral criteria are developed for all organs, the 
Network plans to develop a communication and 
implementation strategy, and expects to begin com- 
municating with referring physicians by the end of 
the 2012/13 fiscal year. 

The Network noted that its organ-specific work- 
ing groups were developing standardized processes 
to remove patients from wait lists, including 
documentation of the reasons for removal, and it 
expected to have the processes for all organ wait 
lists in place by the end of the 2012/13 fiscal year. 
Once that is done, the Network said it plans to peri- 
odically review on a hospital-by-hospital basis the 
number of patients removed from wait lists because 
they die or become too ill for a transplant to see if 
there are any ways to minimize the incidence of 
such cases. 

ALLOCATION OF ORGANS 

Recommendation 4 

To better ensure that organs are allocated in an effi- 
cient and equitable manner, the Trillium Gift of Life 
Network (Network) should: 

• in conjunction with the transplant hospitals, 
review kidney and liver allocations, with a view 
to having one province-wide wait-list (rather 
than up to five regional wait-lists) for each 
organ, so that the highest-priority patient in the 
province, based on clinical evidence, receives the 
first suitable organ available, and transplant 
program sustainability is maintained; 

• have periodic independent reviews conducted 
of organ allocations, to ensure that either the 
highest-priority compatible patient received the 
organ or there was a valid reason for allocating 
the organ to another patient; and 

• provide information to the eight transplant 
hospitals on organs made available but not 
accepted by them, so that the Network and the 



Organ and Tissue Donation and Transplantation 




hospitals can monitor the acceptance rates and 
determine whether any changes are needed to 
the process for offering and accepting organs. 

Status 

At the time of our follow-up, the Network indicated 
that a single wait list for liver transplants was being 
developed and would likely be implemented by 
the end of summer 2012. With respect to persons 
waiting for kidney transplants, a working group 
was evaluating the effect of sharing kidneys among 
the five regions and had agreed in principal to 
move toward a single province-wide wait list. The 
Network expected to have in place by winter 2013 
a system that alternates between allocating one 
kidney locally and one provincially. 

With respect to conducting independent reviews 
to ensure that either the highest-priority patient 
receives the organ, or a valid reason is identified 
for allocating it to another patient, the Network 
noted that a quality committee would be formed to 
consider a system for periodically reviewing organ 
allocation. The Network expected this process to 
be established by the end of the 2012/13 fiscal 
year. In the interim, the Network stated that it has 
improved its chart documentation to better identify 
the rationale for allocating an organ to a patient 
who does not have the highest priority, and is con- 
ducting internal audits of organ allocations. 

The Network expected that changes to its 
current database to track the number of organs 
accepted and declined by the eight transplant 
hospitals would be in place in the 2013/14 fiscal 
year. The Network indicated that, once available, 
it planned to share this information with all trans- 
plant hospitals. 

EFFICIENCY OFTHE ORGAN DONATION 
PROCESS 

Recommendation 5 

To improve the efficiency of the organ donation 
process and avoid delays that may harm the viability 
of donated organs, the Trillium Gift of Life Network 
(Network) should: 



• determine the feasibility of providing transplant 
hospitals with simultaneous electronic access to 
information required to facilitate the physician's 
assessment of the compatibility of the donor and 
a potential recipient, such as the donor's labora- 
tory test results; 

• review the costs and benefits of implementing 
a system capable of tracking the information 
required to oversee the organ donation process, 
including the time taken for each stage of the 
donation process from identification of the 
potential donor to the time of transplant (com- 
pared against target times), and the reasons for 
any delays; and 

• review research on current best practices with 
respect to the use of kidney pumps when trans- 
porting donated kidneys to transplant hospitals 
and track the use of such pumps. 

Further, the Ministry of Health and Long-Term 
Care should review its agreement with the air ambu- 
lance provider, Ornge, and, in conjunction with the 
Network, clarify Ornge's transportation responsibil- 
ities with respect to organ transplantation. 

Status 

At the time of our follow-up, the Network indicated 
that it had conducted a competitive process and 
expected to have a more comprehensive donor- 
management system in place by the end of the 
2012/13 fiscal year. Once implemented, the system 
will enable the sharing of donor data via email with 
transplant hospitals, which helps assess the medical 
suitability of a donor organ. The Network antici- 
pated that further work will be done in the 2013/14 
fiscal year to electronically share donor information 
with the transplant programs at subsequent stages 
of the organ donation process. 

The Network expected that the new donor- 
management system would also facilitate the 
measurement and reporting of more indicators, 
including time spent on each stage of the donation 
process, from identification of a potential donor to 
the time of transplant, and reasons for any delays. 
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The Network said that it completed its review 
of best practices for kidney-pump use and as of 
March 31, 2012, all transplant hospitals have 
agreed to use the pumps to preserve kidneys from 
deceased donors. The Network has also begun 
tracking use of the devices, and has identified pre- 
liminary measures for monitoring and evaluating 
their use, which will be discussed at future meet- 
ings of its Kidney/Pancreas Working Group. 

The Ministry stated that the Network and Ornge 
air ambulance are developing a formal service 
agreement to standardize processes for organ 
transport. The Ministry expected a draft agreement 
would be ready for its review in January 2013. 

TISSUE 

Recommendation 6 

To help ensure that there is an adequate supply of 
quality tissue, such as bones and eyes, to meet the 
needs ofOntarians and reduce reliance on tissue 
purchased from other jurisdictions, the Trillium Gift 
of Life Network (Network) should: 

• increase the number of hospitals required to 
report potential tissue donors to the Network 
and, in conjunction with the hospitals, develop 
more specific clinical triggers (such as age 
criteria) to help hospitals determine which 
patients should be referred to the Network as 
potential tissue donors; 

• review the process of obtaining consent for tissue 
donation, in conjunction with the hospitals, 
with a view to increasing consent rates; and 

• reassess, in conjunction with the tissue banks, 
the screening processes used to determine tissue 
viability so that non-viable tissue is identified as 
quickly as possible. 

Further, the Ministry of Health and Long-Term 
Care, in conjunction with the Network and the tissue 
banks, should: 

• assess the costs and benefits of implementing 
a centralized tissue bank, which would help 
ensure that, after consent is received, tissue 



is recovered, processed, and stored safely and 
efficiently; and 
• consider whether specific funding should be 
provided to offset the costs incurred by hospitab 
and to compensate physicians for their time 
with respect to tissue donation and banking. 

Status 

At the time of our follow-up, the Network said that 
the additional hospitals required to refer potential 
organ donors to the Network also have to refer 
potential tissue donors to the Network. As well, 
the Network stated that it requires reporting of all 
deaths occurring at designated hospitals, but that it 
would consider in the 2012/13 fiscal year the impli- 
cations of moving to more specific clinical triggers 
for reporting potential tissue donors. The Network 
planned to establish a Tissue Working Group, which 
would begin meeting in fall 2012, to increase tissue 
donation through effective donation, recovery and 
banking practices. 

With respect to obtaining consent for tissue 
donation, the Network tracked the consent rates for 
its staff and hospital staff in 2011, and found that 
Network staff obtained an average consent rate of 
50% while untrained hospital staff obtained 4%. 
The Network said it shared this information with 
the hospitals to emphasize the importance of hav- 
ing the Network approach families for consent. The 
Network indicated that in the 2011/12 fiscal year, it 
approached almost 75% of potential tissue donors, 
while hospitals spoke to the rest. 

At the time of our follow-up, the Network noted 
that it was working with various Ontario tissue 
banks to define and implement "exclusion criteria" 
to help identify tissue that is not viable and there- 
fore should not be referred for tissue donation. 
The Network anticipates that this will streamline 
workload and ensure that only relevant referrals 
are made to the tissue banks. 

The Ministry indicated that it was reviewing, in 
conjunction with the Network and other provinces 
and territories, the April 2011 national plan of 
the Canadian Blood Services for organ and tissue 
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donation and transplantation. The plan recom- 
mends a restructured tissue system, including 
nationally centralized tissue banking. At the time 
of our follow-up, the Ministry expected that an 
interprovincial-territorial response to this plan 
would be completed by fall 2012, and therefore it 
was not planning to otherwise assess the costs and 
benefits of a centralized tissue bank. 

The Ministry said it met with the Network and 
physicians who recover tissue donations to discuss 
reimbursements to physicians performing tissue 
recovery. The Ministry concluded that the current 
physician payment schedule included sufficient fees 
for tissue recovery and that it, the Network and the 
Ontario Medical Association had to be more pro- 
active in making physicians aware of these fees. The 
Ministry indicated that hospital funding for tissue 
recovery may need to be considered in future as the 
Health System Funding Strategy is implemented. 

PERFORMANCE MONITORING 

Recommendation 7 

To provide additional assurance that organ and tissue 
transplantation in Ontario is meeting the needs of 
patients safely and efficiently, the Ministry of Health 
and Long-Term Care (Ministry), in conjunction 
with key stakeholders, including the Trillium Gift of 
Life Network, transplant hospitals, and transplant 
physicians, should determine the best structure for 
providing effective oversight for organ and tissue 
transplantation in Ontario, as recommended in 
the 2009 report of the Ministry's Organ and Tissue 
Transplantation Wait Times Expert Panel. As well, 
performance indicators for transplant activity in 
Ontario — such as wait times for transplant by organ, 
number of transplants performed by hospital and 
patient survival rates by hospital — should be estab- 
lished and made publicly available. 



Status 

At the time of our follow-up, the Ministry stated 
that oversight of transplant activities in Ontario 
was a collaborative effort between the Ministry 
and the Network. In this regard, the Network 
established donation and transplant steering com- 
mittees, as well as organ-specific working groups, 
to help implement an integrated structure for 
effective management of organ and tissue donation 
and transplantation in Ontario. The Ministry also 
indicated that, in conjunction with the Network, 
it is reviewing the April 2011 national plan of the 
Canadian Blood Services for organ and tissue dona- 
tion and transplantation. The plan includes recom- 
mendations related to the oversight of donation 
and transplantation, and the Ministry expected that 
an interprovincial-territorial response to this plan 
would be completed by fall 2012. 

At the time of our follow-up, the Network's 
website had data on organ transplant wait lists as 
well as on the number of patients who received a 
transplant, broken down by categories such as age 
and gender. The Ministry noted that the Network is 
consulting with transplant hospitals to establish the 
means and timelines for publicly reporting hospital- 
based transplant statistics, including the number of 
transplants performed by each hospital. However, 
the Ministry indicated that public reporting of other 
data, such as wait time for transplant by organ 
type and life expectancy after transplant, would 
require more time because of factors that included 
the need to develop consistent data definitions and 
implement information technology changes. In 
this regard, the Ministry noted that the Network's 
Transplant Steering Committee and its organ- 
specific working groups were developing standard 
definitions for wait times that it expected would be 
drafted by spring 2013. 
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Background 



A learning environment that is not physically and 
psychologically safe can adversely affect not only 
a student's safety but also his or her motivation to 
learn. The impact of bullying, for example, can be 
severe: victims may have to deal with such issues 
as social anxiety, loneliness, physical ailments, low 
self-esteem, absenteeism, diminished academic 
performance, depression and, in extreme cases, 
thoughts of suicide. A 2009 survey of Ontario 
students in Grades 7 through 12 by the Centre for 
Addiction and Mental Health identified that almost 
one in three students has been bullied at school and 
approximately one-quarter of students have bullied 
others at school. 

Our audit in 2010 focused on the adequacy of 
activities undertaken by the Ministry of Education 
(Ministry) and selected school boards (Durham 
District School Board, Sudbury Catholic District 
School Board and Toronto District School Board) 
to improve the safety of Ontario schools. Our 
work indicated that a number of initiatives had 
been taken to address safety issues in Ontario's 
schools. These initiatives included new legislation, 
teacher training and targeted school safety funding. 
However, neither the Ministry nor the three school 
boards or various schools we visited were collecting 
sufficient information on whether these initiatives 
were having an impact on student behaviour. We 



concluded that better information on the success 
of its initiatives would help the Ministry to allocate 
funding to the areas of greatest need. Some of our 
other key observations were as follows: 

• The Ministry allocated $34 million — about 
two-thirds of its total annual school safety 
funding — to two initiatives focused on 
assisting suspended, expelled, and other high 
risk students. Most of this funding was allo- 
cated based on total board enrolment rather 
than on more targeted factors such as the 
actual number of students needing assistance 

• The percentage of students that had been 
suspended in each board ranged from 1% to 
more than 11% of the student population, 
which may explain why some boards under- 
utilized their funding by as much as 70%. A 
comparison of provincial and school board 
data on suspension rates to a recent anonym- 
ous provincial survey of students suggests, 
however, that school administrators were not 
aware of the extent of serious safety issues 

in some schools, such as the incidence of 
students being threatened or injured with a 
weapon. Most senior safety staff at the school 
boards we visited, as well as administra- 
tors at the schools we visited, said that the 
discrepancy was due to a lack of reporting by 
students, possibly because of fear of reprisals, 
and that more needed to be done to facilitate 
student reporting of incidents. 
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• The Ministry had established requirements 
for school boards and schools pertaining to 
the application of escalating discipline for 
students who had repeatedly violated school 
safety policies. Despite significant differences 
in suspension rates among boards and among 
schools we visited, neither the Ministry nor 
the boards we visited had formally analyzed 
the differences in suspension rates to assess 
whether progressive discipline policies were 
being applied consistently by Ontario's 72 
publicly funded school boards. 

• An evaluation of a program that stations 
police officers in schools identified an 
improvement in relationships between 
students and police. The majority of school 
administrators we interviewed indicated 
that having an officer in the school improved 
school safety and that expansion of such pro- 
grams should be considered. 

We made a number of recommendations for 
improvement and received commitments from the 
Ministry and the school boards that they would 
take action to address our concerns. 



Status of Actions Taken on 
Recommendations 



According to information received from the Min- 
istry and each of the three school boards we visited 
during the audit, progress has been made on imple- 
menting all of the recommendations in our 2010 
Annual Report, with substantial progress made on 
several. For example, the Ministry now monitors 
specific program funding through progress reports 
received from boards and by analyzing data to 
assess activities and how funds were spent to sup- 
port school safety initiatives. In addition, as part 
of the initial investigation into the variations in 
suspension rates within boards and across the prov- 
ince, the Ministry held eight regional data sessions 



across the province for Ontario's publicly funded 
boards in March 2011 and again in spring 2012. 
These regional sessions focused on strategies for 
maintaining consistency in enforcing suspensions 
and expulsions, sharing best practices, and dealing 
with the challenges in designing and implementing 
school safety programs. Also of significance is the 
passing of the Accepting Schools Act, 2012 in June 
2012. This Act encourages stronger actions to create 
a safe and inclusive environment in all schools. 

The status of actions taken on each of our recom- 
mendations is described in the following sections. 

SCHOOL SAFETY INITIATIVES 

Recommendation 1 

To ensure that school safety funding is used effectively 
to achieve program goals to improve school safety, the 
Ministry of Education and, where appropriate, school 
boards should: 

• reconsider the appropriateness of allocating, on 
the basis of enrolment, the majority of school 
safety funding primarily to assist suspended, 
expelled, and other high-risk students, given 
that the ratio of such students to total enrolment 
may vary significantly among school boards; 

• for other specific program funding, ensure that 
the funds are allocated based on identified needs 
and follow up to verify that the funds provided 
are being spent for the intended purpose; and 

• obtain and share information on the success 
of initiatives such as Student Support Leader- 
ship and police officer placements in schools, 
and determine whether a more significant co- 
ordinating role for the Ministry is appropriate 
to enhance their effectiveness. 

Status 

While the main focus of school safety funding has 
been to provide supports for students who have been 
expelled or given long-term suspensions, funds can 
be used for a variety of programs, including those 
to support at-risk students who exhibit behaviours 
that might lead to suspension or expulsion. The 
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Ministry noted that it continues to work with boards 
and its other partners to identify student needs and 
fund specific initiatives to address those needs. The 
three boards we visited in 2010 informed us that 
ministry funding was allocated to each school based 
on factors such as the overall board strategy, board- 
supported initiatives and individual school needs. 
However, the Ministry acknowledged that more 
consistent, reliable information is required to ensure 
that funds are appropriately allocated on the basis of 
student needs. In this regard, the Ministry has been 
engaged in extensive activities designed to enhance 
the quality of student information. 

The Ministry requires detailed reporting on 
all safe schools initiatives on an ongoing basis 
that outlines, in addition to financial information, 
program activities and outcomes. It monitors 
specific program funding through progress reports 
received from boards and by analyzing data to 
assess program activities and how funds were spent 
to support school safety initiatives. The Ministry is 
examining the possibility of conducting spot audits 
to further ensure that the funds provided for these 
initiatives are spent for the purposes intended. 

The Ministry supports partnerships between 
school boards and other community groups 
through its Student Support Leadership Initiative. 
This initiative helps clusters of school boards and 
community agencies provide supports outside the 
classroom that promote positive student behaviour. 
Focusing on the prevention of at-risk behaviours in 
2011/12, the clusters invited leaders of municipal 
services that offer services for children and youth, 
such as parks and recreation and public health, 
to participate. These clusters submitted mid-year 
reports noting their accomplishments and will sub- 
mit final reports in fall 2012. Clusters are also com- 
pleting a self-evaluation of their 2011/12 activities 
that will help them to continue with their activities 
once the financial support from the Student Sup- 
port Leadership Initiative comes to an end. 

Since 2010, the Ministry has gathered and 
shared information on the success of school safety 
initiatives through a variety of other means, such as: 



• hosting the Urban and Priority High Schools 
Symposium for participating schools to share 
information on leading practices and chal- 
lenges (the Ministry intends to produce an 
annual newsletter related to this symposium) ; 

• developing and distributing Promoting a 
Positive School Climate, which provides an 
evidence-based list of practical ideas for Safe 
Schools Teams to consider in their efforts to 
develop and maintain a positive climate for 
students; and 

• establishing a safe schools working group that 
now holds regular teleconferences. 

The Ministry also worked with the Ministry of 
Community Safety and Correctional Services and 
police services partners to negotiate and release a 
revised Provincial Model of the Police/School Board 
Protocol in January 2011. The Ministry has helped 
boards form partnerships with police services by 
providing one-time funding for joint training on the 
revised protocol. In addition, the Ministry of Com- 
munity Safety and Correctional Services provided 
$1.68 million for a Safe Schools Grants program 
intended to: 

• encourage police services, local schools and 
boards to collaborate on various activities; 

• create teams of police, educators, counsellors, 
health-care professionals, parents and stu- 
dents that will work together to help reduce 
violence and bullying; and 

• help police officers become more active and 
engaged in school-related activities. 

The boards we visited during our 2010 audit 
provided further detail on their delivery of the 
Student Support Leadership and other initiatives to 
encourage continued sharing and implementation 
of best practices. One board noted that, to encour- 
age the sharing of best practices across cluster 
members, a number of committees were estab- 
lished that met regularly throughout the school 
year, including a collaborative practices committee 
and a committee to plan and hold partnership 
symposiums. Another board noted that it provided 
release time for its Safe Schools Teams to meet 
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and analyze surveys to identify and address issues 
of concern. The third board initiated a Positive 
Space campaign where each school is to provide a 
designated room or private area within the school 
where students can speak freely to an adult repre- 
sentative. By June 2012, one representative from 
each school was to be trained as a Positive Space 
representative. 

MONITORING COMPLIANCE WITH 
SCHOOL SAFETY REQUIREMENTS 

Recommendation 2 

To promote compliance with all school safety legisla- 
tion and policies designed to provide a safe learning 
environment for Ontario students, the Ministry of 
Education should work with school boards to: 

• monitor compliance with required school safety 
legislation and ministry policies; 

• ensure that schools have functioning Safe School 
Teams in place that include representation from 
all required groups; 

• investigate significant differences in suspension 
rates between school boards and schools to 
assess whether such differences are reasonable 
and to determine whether additional student 
disciplinary guidance is necessary to ensure a 
reasonable level of consistency across the prov- 
ince; and 

• assess whether requiring periodic updates to 
criminal background checks for school staff 
service providers, and volunteers would enhance 
the safety of students in Ontario's schools. 

Status 

The Ministry informed us that it promotes compli- 
ance with school safety legislation, policies and 
initiatives in a variety of ways, including ongoing 
discussions with school boards on reporting, 
monitoring and data-based decision-making. With 
ongoing dialogue and requirements to report on 
specific initiatives, the Ministry expects that boards 
will continue to improve their practices over time. 
The Ministry monitors compliance by analyzing 



data to assess board activities and how funds were 
spent to support school safety. 

In the 2010/11 school year, the Ministry began 
distributing $5 million annually to fund a regionally 
based school board audit function to help ensure 
compliance with school safety legislation, policies 
and initiatives. As of January 31, 2011, audit com- 
mittees had been established in all boards. The 
Ministry developed a risk assessment tool, and 
eight regional internal audit teams were expected 
to complete a risk assessment using this tool by the 
end of the 2011/12 school year. As of April 2012, 
some teams had already completed their risk 
assessment. All school boards were expected to 
have internal audits performed in the 2011/12 
school year, some of which could have been audits 
of school safety initiatives if the region's risk assess- 
ment determined this area to be a priority. 

A memo was sent to all Directors of Education in 
October 2010 reminding boards of the requirement 
to have a functioning Safe Schools Team in place 
with appropriate membership, including the school 
principal and at least one student, parent, teacher, 
non-teaching staff member and community partner. 
In November 2010, the Ministry created the Pre- 
mier's Safe Schools Awards in order to increase the 
profile of these teams. For the 2010/11 school year, 
10 schools were recognized for the exceptional and 
innovative work they were doing to create a safe 
and caring environment. The Ministry is developing 
various ways to showcase Safe Schools Teams that 
have received a Premier's Award, highlighting 
their accomplishments and best practices. Also, the 
Ministry noted that, through a safe schools working 
group teleconference in June 2011, boards shared a 
number of best practices. One of these practices is 
to require schools to report team membership to the 
school board, to help ensure the establishment and 
proper composition of Safe Schools Teams. 

One board reported that it has added a require- 
ment that an equity representative be added to every 
Safe Schools Team to help eliminate biases and 
barriers that may be related to gender, race, religion, 
socio-economic background and other factors. For 
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accountability reasons, the schools report the name 
of the team chair to the board. Also, this board 
advised us that it provides training for Safe Schools 
Teams on topics such as creating a positive school 
climate, its protocol for police-school board partner- 
ships and the use of data to help develop a safety and 
bullying prevention plan. 

In 2011, the Ministry completed an in-depth 
analysis of school board data by examining suspen- 
sion and expulsion rates by school and by board 
and including such student attributes as gender, 
infraction type, length of suspension and the num- 
ber of repeat suspensions. The Ministry found, for 
example, that boys were three times more likely 
to be suspended than girls and more than 80% of 
suspensions were coded as "other," which includes 
such dissimilar violations as fighting, being under 
the influence of drugs, committing uniform infrac- 
tions and being late for school. 

As part of the initial analysis of the variations 
in suspension rates within boards and across the 
province, the Ministry invited each publicly funded 
board to one of eight regional data sessions held 
across the province in March 2011. In-depth analy- 
sis of suspension and expulsion data continues 
and subsequent regional sessions were held in 
spring 2012. The goals of these sessions, which the 
Ministry intends to hold annually, were to work 
toward a more consistent application of legislation, 
regulations and policies in schools and boards and, 
over time, reduce the differences in suspension and 
expulsion rates for students with similar types of 
inappropriate behaviour. 

Discussions at the regional sessions focused on 
strategies for sharing best practices and dealing 
with the challenges in designing and implementing 
programs for suspended and expelled students, 
and on strategies for generating and maintaining 
consistency among school staff in enforcing disci- 
plinary procedures. For example, one board noted 
that it collects suspension rates by school, by area 
and for the whole board mid-year and at the end 
of the school year. This data is then provided to 



area superintendants and school principals, and, 
in camera, to board trustees, for monitoring and 
decision-making purposes. 

A regulation under the Education Act requires 
school boards to collect criminal background checks 
for all employees and, with the exception of school 
bus drivers, for service providers who have direct 
and regular contact with students. School bus driv- 
ers do require criminal background checks, how- 
ever, under the Highway Traffic Act. The Ministry 
of Transportation is expected to remove the licens- 
ing requirement for school bus drivers to obtain 
criminal background checks under the Highway 
Traffic Act effective July 2013. Before that time, the 
Ministry of Education plans to amend its regulation 
under the Education Act to remove the exemption 
of school bus drivers from obtaining criminal 
background checks. In addition to the requirements 
under the regulations, one board noted that it now 
requires criminal background checks for all volun- 
teers, with an annual declaration and an updated 
criminal background check at least every five years. 
Another board advised us that it was in consultation 
with its local police force to determine if ongoing 
updating would enhance safety for students. 

MEASURING AND REPORTING ON 
SCHOOL SAFETY 

Recommendation 3 

To help in its efforts to ensure that students are edu- 
cated in a safe environment, the Ministry of Educa- 
tion should work with school boards to: 

• develop measurable objectives and related 
performance indicators for activities intended to 
improve school safety, and periodically measure 
progress in achieving these objectives; 

• capture data on incidents of inappropriate 
student behaviour and complaints received, in 
addition to the information currently collected 
on suspensions and expulsions, to support the 
assessment of existing initiatives and identify 
areas on which to focus future efforts; 



School Safety 



• conduct school safety surveys to gauge the 
progress achieved in improving school safety at 
the provincial and school board levels; and 

• review existing best practices in Ontario and 
elsewhere that have been found to be effective in 
encouraging students to report serious school 
safety incidents. 

Status 

The Ministry engaged a consultant to develop an 
evaluation framework of indicators for its Safe 
Schools Strategy. The consultant's February 2011 
report to the Ministry included proposed perform- 
ance indicators and data collection strategies. The 
model aims to establish interim accomplishments 
that can indicate whether a program is mak- 
ing progress in the desired direction, given that 
the ultimate goal is likely to take many years to 
accomplish. The consultant based the proposed 
performance indicators on research that included 
current practices in Ontario and other national and 
international jurisdictions, as well as best practices 
in performance measurement. 

The Ministry stated that it is committed to 
implementing a comprehensive evaluation frame- 
work to measure the effectiveness of safe schools 
policies and programs, which would include 
indicators of students' and parents' perceptions of 
school safety. To obtain the data to assess these per- 
ceptions, the Ministry plans to engage an outside 
agency to collect and analyze school survey data, 
starting in the 2012/13 school year, on students' 
and parents' perceptions of school safety. 

The Ministry noted that collecting and analyzing 
aggregate data at the provincial level helps it to 
make legislative, policy and program decisions. The 
Ministry also noted that data collected at the board 
and school level helps boards and schools make 
local decisions on programs and implementation 
strategies that best suit their specific communities. 
In this regard, the Ministry informed us that it 
made amendments to its OnSIS database to capture 
more data from schools on student participation 
in programs for suspended and expelled students. 



Also, in May 2011, the Ministry informed boards 
that they would be required to report data on 
violent incidents to the Ministry beginning in the 
2011/12 school year. The collection of data on 
student participation in these programs and on 
incidents will form the basis for the measurement 
of some of the proposed performance indicators for 
the Safe Schools Strategy. 

In November 2010, the Ministry spent $3.1 mil- 
lion to develop and distribute resource guides to 
promote a positive school climate. Receipt of this 
funding was contingent upon the school's having 
undertaken a survey to assess whether the climate 
in the school is inclusive and accepting of all pupils, 
including pupils of any race, gender, creed, sexual 
orientation or disability. 

All three boards reported on the use of school 
surveys. One board noted that a pilot survey was 
given to a number of schools in the spring of 2011 
and then extended to all of its schools in the fall. 
The boards noted that the surveys were used for 
multiple purposes, such as to identify areas of 
concern that needed to be addressed, to support 
the development of safety and bullying prevention 
plans and to assist in the understanding of the 
school climate in order to improve program plan- 
ning at the school level. 

The Ministry was also involved in the develop- 
ment of legislation that requires board multi-year 
plans to include goals that promote a positive 
school climate and bullying prevention. The legisla- 
tion also requires each board to establish a bullying 
prevention plan that will include procedures for 
reporting the incidence of bullying and the range 
of disciplinary actions a principal may take against 
a student involved in bullying. The legislation also 
establishes strategies for protecting a person who 
witnesses an act of bullying, reports bullying or pro- 
vides information during a bullying investigation. 
This legislation, the Accepting Schools Act, 2012, 
which amended the Education Act, received royal 
assent in June 2012. 
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SCHOOL SAFETY TRAINING 

Recommendation 4 

To build on the steps taken to date to ensure that 
school staff are adequately trained to deal with school 
safety issues, the Ministry of Education should work 
with school boards to assess whether school safety 
training delegated to schools is of sufficient depth to 
meet the needs of school staff. 

Status 

The Ministry informed us that, in consultation with 
school boards, it continues to build on the training 
that it has provided over the last several years with 
respect to safe and inclusive schools. The Ministry 
also noted that it is working closely with boards to 
meet the needs that boards identify in the areas of 
training and staff development. 

The Ministry continues to share, and create 
opportunities to share, best practices on how to 
train staff on an ongoing basis. For example, at 
the regional sessions it was noted that e-learning 



and the analysis of various scenarios are effect- 
ive methods of training staff within boards and 
schools. In May 2011, the Ministry also funded 
Toronto Police Services to produce a training video 
on the revised police-school board protocol and 
provided funding for joint training of school board 
and police services staff. 

All three boards we visited during our audit 
advised us that they had systems in place to track 
and provide professional development oppor- 
tunities for board and school staff. One board 
developed a central tracking tool that staff use to 
register for professional learning opportunities 
found within the board. A second board noted that 
it provides its schools with consistent school safety 
training packages related to various initiatives. The 
third board noted that it provides ongoing training 
and resources to the Safe Schools Teams in order to 
support the planning that is being done with school 
climate surveys to extend the development of a 
positive climate within its schools. 
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Overview of the Advertising 
Review Function 



The government of Ontario is a major Canadian 
advertiser, spending tens of millions of dollars 
each year to place advertisements on television, 
radio, the Internet, billboards and in newspapers 
and magazines. 

The Government Advertising Act, 2004 (Act) took 
effect in December 2005 after two years of debate 
in the Legislature — and several years of discussion 
prior to that — as legislators questioned the appro- 
priateness of a government spending public funds 
on advertising that could be considered to further 
its partisan interests. 

The main intent of the Act is to prohibit govern- 
ment advertising that could be seen as promoting 
the governing party's political interests by fostering 
a positive impression of the government or a nega- 
tive impression of any group or person critical of 
the government. The Act mandates the Auditor 
General to review and approve most government 
advertisements before they run to ensure that 
they comply with the Act's prohibition on partisan 
advertising and its other standards. The full text of 
the Act can be found at www.e-laws.gov.on.ca. 

This chapter satisfies the legislative require- 
ments in the Act as well as in the Auditor General 
Act to report annually to the Legislative Assembly 
on the work we have done over the past fiscal year 
to ensure adherence to the Act. 



Under the Act, the Auditor General is responsible 
for reviewing specified types of government adver- 
tisements to ensure they meet legislated standards. 
Above all, they must not contain anything that is, 
or could be interpreted as being, primarily partisan 
in nature. 

The Act outlines standards each advertisement 
must meet and states that "an item is partisan if, 
in the opinion of the Auditor General, a primary 
objective of the item is to promote the partisan pol- 
itical interests of the governing party." 

The Act also provides the Auditor General with 
the discretionary authority to consider additional 
factors in determining whether a primary objective 
of an item is to promote the partisan interests of the 
governing party (see the "Other Factors" section 
later in this chapter) . 

WHAT FALLS UNDER THE ACT 

The Act applies to advertisements that govern- 
ment offices — specifically, government ministries, 
Cabinet Office and the Office of the Premier — pro- 
pose to pay to have published in a newspaper or 
magazine, displayed on a billboard, or broadcast 
on radio or television. It also applies to printed 
matter that a government office proposes to pay 
to have distributed to households in Ontario using 
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unaddressed bulk mail or another method of bulk 
delivery. Advertisements meeting any of these def- 
initions are known as "reviewable" items and must 
be submitted to my Office for review and approval 
before they can run. 

The Act excludes from review job advertise- 
ments and notices to the public required by law. 
Advertisements on the provision of goods and ser- 
vices to a government office, and those on urgent 
matters affecting public health or safety, are also 
excluded. 

Although the following are not specifically 
excluded, we have come to a mutual understanding 
with the government that they are not subject to 
the Act: 

• online advertising; and 

• brochures, newsletters, consultation docu- 
ments, reports and other similar materials or 
publications. 

The Act requires government offices to submit 
every reviewable item to our Office. The govern- 
ment office cannot publish, display, broadcast, 
distribute or disseminate the submitted item until 
the head of that office, usually the deputy minister, 
receives notice, or is deemed to have received 
notice, that the advertisement has been approved. 

The Auditor General's Office, by regulation, has 
seven business days to render its decision. If we do 
not give notice within this time, the government 
office is deemed to have received notice that the 
item meets the standards of the Act, and the item 
may be run. 

If my Office notifies the government office that 
the item does not meet the standards, the item may 
not be used. However, the government office may 
submit a revised version of the rejected item for 
another review. As with the first submission, my 
Office has seven days to render a decision. Under 
the Act, all decisions of the Auditor General are 
final. 

Approval of an advertisement is valid for one 
year, although my Office can rescind an approval 
if we determine that new circumstances leave the 
impression that the ad has become partisan. 



A pre-review is also available to government 
offices wishing us to examine an early version of an 
ad. This can be a script or storyboard, provided that 
it reasonably reflects the item as it is intended to 
appear when completed. Pre-reviews help limit the 
time and money spent to develop ads containing 
material that could be deemed objectionable under 
the Act. A pre-review is strictly voluntary on our 
part and is outside the statutory requirements of 
the Act. 

If material submitted for pre-review appears 
to violate the Act, we provide an explanation to 
the government office. If it appears to meet the 
standards of the Act, we so advise the government 
office. However, before the advertisement can be 
used, the government office must submit it in fin- 
ished form so we can review it to ensure that it still 
meets the standards of the Act. 

STANDARDS FOR PROPOSED 
ADVERTISEMENTS 

In conducting its review, the Auditor General's 
Office determines whether the proposed advertise- 
ment meets the standards of the Act, which are: 

• The item must be a reasonable means of 
achieving one or more of the following 
objectives: 

• to inform the public of current or proposed 
government policies, programs or services; 

• to inform the public of its rights and 
responsibilities under the law; 

• to encourage or discourage specific social 
behaviour in the public interest; and/or 

• to promote Ontario, or any part of the prov- 
ince, as a good place to live, work, invest, 
study or visit, or to promote any economic 
activity or sector of Ontario's economy. 

• The item must include a statement that it is 
paid for by the government of Ontario. 

• The item must not include the name, voice or 
image of a member of the Executive Council 
(cabinet) or a member of the Legislative 
Assembly (unless the primary target audience 
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is located outside Ontario, in which case the • 
item is exempt from this requirement). 

• The item must not have a primary objective 
of fostering a positive impression of the • 
governing party, or a negative impression of a 
person or entity critical of the government. 

• The item must not be partisan; that is, in the 
opinion of the Auditor General, it cannot have 
as a primary objective the promotion of the 
partisan interests of the governing party. • 



OTHER FACTORS 

In addition to the specific statutory standards 
above, the Act allows the Auditor General to 
consider additional factors to determine whether 
a primary objective of an item is to promote the 
partisan interests of the governing party. In gen- 
eral, these additional factors relate to the overall 
impression conveyed by the ad and how it is likely 
to be perceived. Consideration is given to whether 
it includes certain desirable attributes and avoids 
certain undesirable ones, as follows: 

• Each item should: 

• contain subject matter relevant to govern- 
ment responsibilities (that is, the govern- 
ment should have direct and substantial 
responsibilities for the specific matters 
dealt with in the item) ; 

• present information objectively, in tone and 
content, with facts expressed clearly and 
accurately, using unbiased and objective 
language; 

• emphasize facts and/or explanations, 
rather than the political merits of propos- 
als; and 

• enable the audience to distinguish between 
fact on the one hand and comment, opinion 
or analysis on the other. 

• Items should not: 

• use colours, logos and/or slogans com- 
monly associated with the governing party; 

• inappropriately personalize (for instance, 
by attacking opponents or critics) ; 



directly or indirectly attack, ridicule, criti- 
cize or rebut the views, policies or actions 
of those critical of the government; 
intentionally promote, or be perceived as 
promoting, political-party interests (to this 
end, consideration is also given to the tim- 
ing of the message, the audience it is aimed 
at and the overall environment in which 
the message will be communicated) ; 
deliver self-congratulatory or image- 
building messages; 

• deal with matters such as a policy proposal 
where no decision has been made, unless 
the item provides a balanced explanation of 
both the benefits and the disadvantages; 

• present pre-existing policies, products, 
services or activities as if they were new; or 

• use a uniform resource locator (URL) to 
direct readers, viewers or listeners to a web 
page with content that may not meet the 
standards of the Act (see "Websites" in the 
following section). 

OTHER REVIEW PROTOCOLS 

Since taking on responsibility for the review of 
government advertising, my Office has tried to 
clarify, in co-operation with government offices, 
areas where the Act is silent. What follows is a brief 
description of the main areas that have required 
clarification over the years. 

Websites 

Although websites are not specifically reviewable 
under the Act, we believe that a website, Quick 
Response Code or similar linkage used in an adver- 
tisement is an extension of the ad. Following discus- 
sions with the government in 2005, we came to an 
agreement that the first page, or "click," of a website 
cited in a reviewable item would be included in our 
review. We consider only the content of the first 
click, unless that first click is a gateway page, in 
which case we review the next page. We examine 
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this page for any content that may not meet the 
standards of the Act. For example, the page must 
not include a minister's name or photo, any self- 
congratulatory messages or any content that attacks 
the policies or opinions of others. 

Third-party Advertising 

Government funds provided to third parties are 
sometimes used for advertising. The government 
and my Office have agreed that third-party adver- 
tising must be submitted for review if it meets all 
three of the following criteria: 

• a government office provides the third party 
with funds intended to pay part or all of the 
cost of publishing, displaying, broadcasting or 
distributing the item; 

• the government grants the third party permis- 
sion to use the Ontario logo or another official 
provincial visual identifier in the item; and 

• the government office approves the content of 
the item. 

Social Media 

Social media was in its infancy when the Act came 
into effect in 2005. However, its use has grown 
exponentially in recent years. This year, for the 
first time, our Office received ads for approval with 
icons pointing to the government's presence on 
social-media sites. Although the Act is silent on this, 
we reached an agreement with the government that 
our Office will perform an initial scan of any social- 
media channel cited in an ad to ensure that there 
are no partisan references. However, we recognize 
that content on these networks changes frequently 
and can at times be beyond the control of the gov- 
ernment office. 

Event/Conference Program Advertisements 

Government advertisements sometimes appear in 
programs and other materials distributed at public 
events such as conferences and trade shows. Our 



Office took the view that these ads should be sub- 
ject to the Act because they serve a purpose similar 
to magazines and other print media. Often, these 
ads are run free of charge because the government 
has financially sponsored the event. We therefore 
consider that the government indirectly paid for the 
"free" advertisement. The government has agreed 
with this approach, and these items must be sub- 
mitted for review. 

Government Recruitment Advertisements 

As previously noted, the Act excludes job adver- 
tisements from review. We have interpreted this 
exemption as applying to advertising for specific 
government jobs, but not to broad-ranging generic 
recruitment campaigns. The government has 
agreed with our interpretation and, as a result, 
generic recruitment campaigns must be submitted 
to my Office for review. 



External Advisers 



Under the Auditor General Act, the Auditor General 
can appoint an Advertising Commissioner to help 
to fulfill the requirements of the Government Adver- 
tising Act, 2004. Instead, my Office has engaged 
a number of external advisers to assist us in the 
review of submissions. The following advisers 
provided services to my Office during the 2011/12 
fiscal year: 

• Rafe Engle (J.D., L.L.M.) is a Toronto lawyer 
specializing in advertising, marketing, com- 
munications and entertainment law for a 
diverse group of clients in the for-profit and 
not-for-profit sectors. He also acts as the out- 
side legal counsel for Advertising Standards 
Canada, and as Chair of its Advertising Stan- 
dards Council. Before studying law, Mr. Engle 
acquired a comprehensive background in 
media, advertising and communications while 
working in the advertising industry. 




• Jonathan Rose is Associate Professor of 
Political Studies at Queen's University. He is 
a leading Canadian academic with interests 
in political advertising and Canadian politics. 
Professor Rose has written a book on govern- 
ment advertising in Canada and a number of 
articles on the way in which political parties 
and governments use advertising. 

• Joel Ruimy is a communications consultant 
with three decades of experience as a journal- 
ist, editor and producer covering Ontario and 
national politics in print and television. 

• John Sciarra is the former director of oper- 
ations in my Office. He was instrumental in 
implementing our advertising review function 
and overseeing our review responsibilities 
until his retirement in 2010. 

These advisers provided valuable assistance in 
our review of government advertising this past year. 



Advertising Review Activity, 
2011/12 



RESULTS OF OUR REVIEWS 

During the 2011/12 fiscal year, we reviewed 565 
individual advertising items in 121 final submis- 
sions, with a total value of $34.8 million. This 
includes 38 ads, worth $202,000, that had not been 
submitted to our Office for review, but should have 
been. This compares to 165 submissions, compris- 
ing 1,082 individual ads, with a total value of more 
than $50 million last year. 

We gave our decision within the required 
seven business days in all cases. The length of 
time required for a review and decision can vary, 
depending on the complexity of the ad and on 
other work priorities. Nevertheless, the average 
turnaround time during the past fiscal year was 
3.2 business days. 

We also examined eight pre-review submissions 
that contained 11 ads at a preliminary stage of 
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development. Because pre-reviews are voluntary 
on our part and outside the statutory require- 
ments of the Act, they are second in priority to 
finished items. We nonetheless make every effort 
to complete them within a reasonable time. The 
average turnaround time for these submissions was 
5.6 business days. 

Of all the final submissions received in the 
2011/12 fiscal year, we found two in violation: 

• A radio campaign promoting the availability 
of free vaccinations for the rotavirus was 
rejected because it loosely resembled a Liberal 
Party of Ontario commercial that ran prior to 
the 2011 election on the medical screening of 
newborn infants. The Ministry of Health and 
Long-Term Care subsequently reworked the 
ad and resubmitted it, and we approved it. 

• A print and radio campaign about agricultural 
risk-management programs was rejected 
because it violated section 6(1)5 of the Act, 
which says that an ad must not have as a pri- 
mary objective to "foster a positive impression 
of the governing party." After quickly rework- 
ing the campaign, the Ministry of Agriculture, 
Food and Rural Affairs resubmitted it and we 
approved it. 

We also rescinded approval for three digital 
video and television ads that were part of a Ministry 
of Health and Long-Term Care campaign on med- 
ical wait times after the Liberal Party of Ontario 
released an ad with strikingly similar visuals on the 
same subject prior to the 2011 election. 

We also cited the Ministry of Finance for a viola- 
tion of the Act after we found that the campaign 
for Ontario Savings Bonds, which we had earlier 
approved, contained a change to the approved web 
page that included a reference to the "McGuinty 
government." This contravenes section 6(1)3 of the 
Act, which says ads "must not include the name, 
voice or image of a member of the Executive Coun- 
cil or a member of the Assembly." The Ministry 
quickly corrected it. 
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We also noted contraventions of the Act — adver- 
tisements that ran without having first been submit- 
ted to us for review. They are as follows: 

• The Ministry of the Attorney General advised 
us that it ran an ad in a publication for lawyers 
about a new mailing address for the Office of 
the Children's Lawyer without first submitting 
it for review. Had the ad been submitted, our 
Office would have approved it. 

• The Ministry of Community and Social Servi- 
ces advised us that it had run an ad over the 
last six years about ministry services available 
to Toronto's francophone community without 
first submitting it for review. Had the ad been 
submitted, our Office would have approved it. 

• The Ministry of Economic Development and 
Innovation advised us that its advertising 
agency inadvertently released an ad for publi- 
cation although it had not yet been submitted 
to our Office for review. This ad was part of 

a large international campaign whose other 
elements were submitted to and approved by 
our Office. The Ministry and agency advised 
us they have since adopted new procedures 
to ensure this does not happen again. The ad 
in question was subsequently submitted and 
approved by our Office. 

• We discovered through our expenditure- 
gathering process that the Ministry of Natural 
Resources ran 34 radio ads that it did not 
submit to our Office for review. The Ministry 
explained that these ads were about urgent 
matters regarding forest fires and therefore 
exempt from review under subsection 2(5) of 
the Act. We reviewed the ads and determined 
that 12 of them did not qualify for the exemp- 
tion as they focused on fire prevention which, 
in our opinion, did not meet the intent of the 
Act with respect to the exclusion of urgent 
messages. While our Office prefers to be 
notified prior to the government running an 
ad on an urgent matter, it may not always be 
practical. We have therefore established an 
urgent-advertising process with the Ministry, 



whereby our Office will be provided copies on 
a monthly basis of any urgent-advertising run. 
• The Ministry of Tourism, Culture and Sport 
ran 23 ads in various media about Huronia 
Historical Parks and Fort William Historical 
Park without first submitting them to our 
Office. These attractions are part of the Min- 
istry and therefore covered by the Act. These 
contraventions occurred in April, May and 
June 2011. Since we raised this issue with the 
Ministry in July 2011, it has been submitting 
such advertising for review. 

OTHER MATTERS 

Online Advertising 

Online advertising has become an integral part of 
most marketing campaigns. Many of the govern- 
ment's large advertising campaigns include an 
online component and some campaigns even run 
entirely online. The Act does not cover online 
advertising. In the past, we have seen government 
online campaigns that would have been in violation 
of the Act if they had been submitted to our Office 
for review. In the 2011/12 fiscal year, the govern- 
ment spent more on Internet advertising than it did 
for advertising on television (excluding production 
costs) . As the government's online advertising 
increases, we believe its exemption has become a 
significant loophole in the legislation that should 
be addressed if the intent of the Act is to be met 
for all government advertising. Our expenditure 
reporting at the end of this chapter does not include 
any Internet costs. We believe the time has come 
for the government to amend the Act to include 
Internet advertising. 

Revised Government Advertising 
Guidelines 

In the fall, our Office released updated Advertising 
Review Guidelines. The revised Guidelines provide 
government offices with an overview of the Act, 
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explain our processes for submitting advertise- 
ments for review, and provide clarification on cer- 
tain sections of the Act. They also reflect evolving 
practices adopted since the Act took effect seven 
years ago. We have posted the Guidelines on our 
website to provide ready access to staff in govern- 
ment offices. 



Expenditures on 
Advertisements and Printed 
Matter 



The Auditor General Act requires that the Auditor 
General report annually to the Legislative Assembly 
on expenditures for advertisements and printed 
matter reviewable under the Government Advertis- 
ing Act, 2004. 

Figure 1 contains expenditure details of adver- 
tising campaigns reported to us by each ministry. 
In order to test the completeness and accuracy of 
the reported advertising expenditures, my Office 
reviewed selected payments to suppliers of adver- 
tising and creative services and their supporting 
documentation at selected ministries. We also per- 
formed certain compliance procedures with respect 
to the requirements of sections 2, 3, 4 and 8 of the 
Government Advertising Act, 2004. These deal with 
submission requirements and prohibition on the use 
of items pending the Auditor General's review. We 
found no matters of concern in our review work. 
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Figure 1: Expenditures for Reviewable Advertisements and Printed Matter under the 
Government Advertising Act, 2004, April 1, 2011-March 31, 2012 

Source of data: Ontario government offices 





#of 


#of 


Third-party Costs ($) 




Ministry/ Campaign Title 


Submissions 


Items 


Agency Fees 


Production 


Talent 


1 


Bulk Mail 


Other 


Aboriginal Affairs 


Infrastructure Signs 


2 


22 




2,038 








Agriculture, Food and Rural Affairs 


Foodland Ontario 1 
















Good Things Grow in Ontario 


4 


64 


177,623 


109,286 


227,789 




158 


Invest in Ontario 1 
















Risk Management Programs 4 


2 


12 












Attorney General 


Notice to the Legal Profession 


1 


1 












Notice to the Legal Profession 3 




1 












Citizenship and Immigration 


Global Experience Ontario 


2 


2 










40 


Order of Ontario 


1 


23 


3,488 








220 


Remembrance Day 


1 


8 


1,111 








184 


Community and Social Services 


Social Service Contact Numbers 3 




1 












Community Safety and Correctional Services 


Arrive Alive 


1 


1 












Cyber Safety 


1 


1 




18,000 








RIDE 1 
















RIDE 2 


1 


8 






27,415 




612 


Economic Development and Trade 


Business Immigration 


3 


13 


13,912 


3,400 






401 


Business Immigration 1 
















Go North 1 






-340 


959 






191 


Invest Ontario 


4 


26 


54,629 


3,146 


6,000 




17 


Invest Ontario 1 






100,194 


5,892 


6,000 




4,492 


Ontario Exports 


1 


1 


528 










Ontario Exports 1 






11,106 


72 








Your Next Big Idea 2 


3 


27 


569,148 


236,917 






520 


Your Next Big Idea 6 


1 


1 












Education 


Full-day Kindergarten 


2 


13 


15,443 


9,686 


30,422 






Full-day Kindergarten 1 

















1. Reported in 2011, but more costs in 2012. 

2. Reported in 2012, but more costs in 2013. 

3. Contravention-ad was not submitted for review, but reported by Ministry. 

4. Violation-ad was reviewed, but did not meet the required standards, then resubmitted and approved. 
6. Contravention-ad was published before being reviewed, then submitted and approved. 
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Media Costs ($) 


Ad Value* 


Campaign 


TV 


Radio 


Print 


Out-of-Home* 


($) 


Total ($) 














2,038 




2,954,708 


329 




20,581 




2,975,618 


322,181 


740,793 


380,823 


517,474 




2,476,127 






7,938 






7,938 




92,648 


40,163 






132,811 








1,827 






1,827 






3,574 






3,574 








1,250 




248 


1,538 






140,582 






144,290 






30,174 






31,469 








2,052 






2,052 








1,831 






1,831 


55,334 










73,334 


-367 










-367 a 


108,055 










136,082 




12,463 




30,683 






60,859 






-10,202 






-10,202 a 






-5,077 




500 


-3,767 a 






122,330 






186,122 


-1,941 




71,910 






186,547 










5,500 


6,028 






11,439 






22,617 






5,125,369 


336,034 




6,267,988 
















953,954 










1,009,505 


-17,728 




940 






-16,788 a 



* Out-of-Home advertising includes, for example, billboards and transit posters. 

t Ad Value denotes the value of an ad space provided to government offices at no cost, often where the government has provided funding for a related event, 
a. Negative total due to media credits being applied. 
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#of 


#of 


Third-party Costs ($) 




Ministry/ Campaign Title 


Submissions 


Items 


Agency Fees 


Production 


Talent 


1 


Bulk Mail 


Other 


Energy 


Long-term Energy Plan 1 
















Environment 


Protecting Ontario's Drinking Water 


1 


1 












Finance 


Children's Activity Tax Credit 1 
















Ontario Budget 1 






6,525 


13,775 






3,115 


Ontario Savings Bonds 7 


3 


32 


403,260 


491,638 


92,414 




10,465 


Government Services 


Licence Plate Sticker Renewal 


2 


3 


4,720 


14,137 


9,046 






Postal Strike 


1 


2 










771 


ServiceOntario 1 
















ServiceOntario Brand Awareness 


3 


32 


165,012 


493,875 


46,287 




3,708 


ServiceOntario Offices 


3 


7 










60 


ServiceOntario Offices 2 


1 


4 












Health and Long-Term Care 


Breast Cancer Screening Expansion 


3 


24 




77,863 






18 


Diabetes 


4 


28 




2,300 






4,000 


Health Care Options 


1 


11 


2,756 








3,252 


Health Care Options 5 
















Health Care Options 1 
















Healthy Smiles Ontario 


1 


2 


5,270 


197,874 








Immunization - Rotavirus Vaccine 4 


2 


4 


4,250 


8,590 


9,625 




567 


MedsCheck for Diabetes Patients 1 
















Public Information Session 


1 


1 


350 


1,010 


1,000 




50 


Seasonal Flu 


4 


25 


43,478 


83,295 


121,260 




1,961 


Seasonal Flu 1 
















Stroke Warning Signs 2 


2 


2 


7,235 


25,241 






3,417 


Trillium Gift of Life Network 


4 


18 


79,231 


312,430 


82,628 




3,102 


Health Promotion and Sport 


Diabetes 1 
















World Junior Baseball 1 
















Labour 


Safe at Work Ontario 


1 


1 












Employment Standards 1 

















1. Reported in 2011, but more costs in 2012. 

2. Reported in 2012, but more costs in 2013. 

4. Violation-ad was reviewed, but did not meet the required standards, then resubmitted and approved. 

5. Reported in 2011, but more costs in 2012-approval withdrawn. 

7. Changes to the website after approval violated the Act, then corrected by Ministry. 
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Media Costs ($) 


Ad Value* 


Campaign 


TV 


Radio 


Print 


Out-of-Home* 


($) 


Total ($) 




488,492 


1,218,136 








1,706,628 












100 


100 






-2,077 


264,334 






262,257 












23,415 


791,971 


157,326 


671,932 






2,619,006 






9,865 


8,742 


6,037 




52,547 






93,932 






94,703 






17 






17 


1,569,587 




361,306 






2,639,775 






5,416 






5,476 




















764,796 






842,677 






860,062 




3,250 


869,612 




112,022 








118,030 


1,583,001 










1,583,001 








186,614 




186,614 






196,912 






400,056 




375,880 








398,912 




-2,379 








-2,379 a 






7,511 






9,921 


1,351,051 






271,873 




1,872,918 




5,622 








5,622 












35,893 


1,244,106 




20,769 


1,677,192 




3,419,458 








141,082 






141,082 










2,400 


2,400 












1,195 


1,195 






-7,260 






-7,260 a 



* Out-of-Home advertising includes, for example, billboards and transit posters. 

f Ad Value denotes the value of an ad space provided to government offices at no cost, often where the government has provided funding for a related event, 
a. Negative total due to media credits being applied. 
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Ministry/ Campaign Title 


Submissions 


Items 


Agency Fees 
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Ontario Parks 1 








375 








Ontario Parks 2 


2 


3 












Outdoors Card Renewal 


1 


1 




169 








Shoreline Property Notices 


2 


2 












Species at Risk 


1 


1 












Wildlife Management 


2 


2 




375 








Wildfire Prevention 3 




12 












Northern Development, Mines and Forestry 


Northern Ontario Energy Credit 1 
















Northern Ontario Heritage Fund 1 
















Revenue 


Tax Credits 1 






101,479 


91,283 


48,254 


300,560 


18,634 


Tourism, Culture and Sport 


Fort William Historical Park 


11 


20 












Fort William Historical Park 3 




16 












Huronia Historical Parks 


17 


31 




2,615 






745 


Huronia Historical Parks 3 




7 




7,920 








Training, Colleges and Universities 


Employment Ontario 


2 


2 












Post-secondary Awareness and 
Public Education 1 
















Study in Ontario 2 


1 


1 












Transportation 


Drive Safe 


1 


1 












Veterans Graphic Licence Plates 


1 


3 


8,000 


770 


6,729 






Total 


121 


565 


1,778,408 


2,220,285 


714,869 


300,560 


64,780 



1. Reported in 2011, but more costs in 2012. 

2. Reported in 2012, but more costs in 2013. 

3. Contravention-ad was not submitted for review, but reported by Ministry. 
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40,600 






40,975 






















14,560 


14,729 






1,506 






1,506 






1,108 






1,108 










12,860 


13,235 




78,024 








78,024 






102,113 








102,113 










6,024 


6,024 




1,144,300 




704,151 






2,408,661 






8,722 


32,739 


64,113 




105,574 




20,330 


67,870 






88,200 


35,342 


29,918 


42,569 






111,189 






1,469 


20,650 




30,039 












50,000 


50,000 


-8 




-10,374 






-10,382 a 






15,228 




1,200 


16,428 








5,000 






5,000 


535,933 




10,465 






561,897 


13,146,014 


2,947,272 


10,379,658 


3,107,489 


106,815 



* Out-of-Home advertising includes, for example, billboards and transit posters. 

t Ad Value denotes the value of an ad space provided to government offices at no cost, often where the government has provided funding for a related event, 
a. Negative total due to media credits being applied. 



Chapter 6 



The Standing Committee 
on Public Accounts 



Appointment and 
Composition of the 
Committee 




Members of the Standing Committee on Public 
Accounts (Committee) are appointed by a motion of 
the Legislature. The number of members from any 
given party reflects that party's representation in the 
Legislative Assembly All members except the Chair 
are entitled to vote on motions, while the Chair may 
vote only to break a tie. The Committee is normally 
established for the duration of the Parliament, from 
the opening of its first session immediately follow- 
ing a general election to its dissolution. 

Following the October 2011 election of a minor- 
ity Liberal government, agreement was reached on 
the Committee's membership in February 2012. In 
accordance with the Standing Orders of the Assem- 
bly, the Committee was appointed on February 23, 
2012, with the following members: 

Norm Miller, Chair, Progressive Conservative 

Toby Barrett, Vice-chair, Progressive 
Conservative 

France Gelinas, New Democrat 

Phil McNeely, Liberal 

Reza Moridi, Liberal 

Jerry Ouellette, Progressive Conservative 

Liz Sandals, Liberal 

Jagmeet Singh, New Democrat 

David Zimmer, Liberal 



On September 10, 2012, in accordance with the 
February 23 motion, the Committee was dissolved. 
At the time this report was prepared, a motion 
had not yet been passed appointing the committee 
members for the fall 2012 session. The Legislature 
was prorogued on October 15, 2012. 

The Committee is empowered to review and report 
to the House its observations, opinions and recom- 
mendations on reports from the Auditor General 
and on the Public Accounts, which are deemed 
to have been permanently referred to the Com- 
mittee as they become available. The Committee 
examines, assesses and reports to the Legisla- 
tive Assembly on a number of issues, including 
the economy and efficiency of government and 
broader-public-sector operations and the effective- 
ness of programs in achieving their objectives. The 
Committee typically holds a number of hearings 
throughout the year relating to matters raised 
in our Annual Report or our special reports and 
presents its observations and recommendations to 
the Legislative Assembly. Under sections 16 and 17 
of the Auditor General Act, the Committee may also 
request that the Auditor General examine any mat- 
ter in respect of the Public Accounts or undertake a 
special assignment on its behalf. 



420 



AUDITOR GENERAL'S ADVISORY ROLE 
WITH THE COMMITTEE 

In accordance with section 16 of the Auditor Gen- 
eral Act, the Auditor General and senior staff attend 
all committee meetings to assist the Committee 
with its reviews and hearings relating to our Annual 
Report, Ontario's Public Accounts and any special 
reports issued by our Office. 





Committee Procedures and 
Operations 



The Committee may meet weekly when the Legisla- 
tive Assembly is sitting, and, with the approval of 
the House, at any other time of its choosing. All 
meetings are open to the public except for those 
dealing with the Committee's agenda or the prep- 
aration of its reports. All public committee proceed- 
ings are recorded in Hansard, the official verbatim 
report of government debates, speeches and other 
Legislative Assembly proceedings. 

The Committee identifies matters of interest 
from our Annual Report or our special reports and 
conducts hearings on them. It typically focuses on 
reports from the value-for-money chapter of our 
Annual Report for review. For example, each of the 
three political parties normally selects three audits 
or other sections from our Annual Report for the 
Committee to review. 

At each hearing, the Auditor General, along with 
the Committee's researcher, briefs the Committee 
on the applicable report section and the responses 
to our findings and recommendations from the min- 
istry, Crown agency or organization in the broader 
public sector that was the subject of the audit. The 
Committee typically asks senior officials from the 
auditee(s) to appear at the hearings and respond to 
questions from committee members. Because our 
Annual Report deals with operational, administra- 
tive and financial rather than policy matters, minis- 
ters are rarely asked to attend. Once its hearings are 



The Standing Committee on Public Accounts 




completed, the Committee provides its comments 
and recommendations to the Legislative Assembly. 

The Clerk of the Committee also requests that 
those auditees that were not selected for hearings 
update the Committee on what actions they are 
taking to address the concerns raised and recom- 
mendations made in our reports. 

MEETINGS HELD 

The Committee met 20 times during the February 
2012-September 2012 period. Almost all of these 
meetings were related to our Office's March 2012 
special report, Ornge Air Ambulance and Related 
Services, and included a number of sessions where 
witnesses were called and questioned. The remain- 
ing meetings were either administrative in nature 
or related to a motion filed by one of the commit- 
tee members requesting that our Office examine 
certain Ontario Power Authority contracts. In its 
final meeting on September 5, 2012, the Committee 
passed a related motion requesting that our Office 
examine the contract between the Ontario Power 
Authority and Greenfield South Power Corpora- 
tion/Eastern Power Limited regarding the cost to 
taxpayers of the cancelled Mississauga gas plant 
project. We plan to issue a Special Report on the 
results of this work in 2013. 

REPORTS 0FTHE COMMITTEE 

The Committee issues reports and letters on its work 
for tabling in the Legislative Assembly. These reports 
and letters summarize the information gathered 
by the Committee during its meetings and include 
the Committee's comments and recommendations. 
Once tabled, all committee reports and letters are 
publicly available through the Clerk of the Commit- 
tee or online at www.ontla.on.ca. 

Committee reports typically include recommen- 
dations and request that management of the Min- 
istry, agency or broader-public-sector organization 
provide the Committee Clerk with responses within 
a stipulated time frame. Our Office reviews these 
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recommendations and responses, and we take them 
into consideration in any subsequent follow-up sec- 
tion or audits of that operational area. 

Since its appointment in February 2012, the 
Committee focused almost exclusively on matters 
relating to our special report, Ornge Air Ambulance 
and Related Services, and these hearings were still 
ongoing when the Committee was dissolved in Sep- 
tember 2012. No committee reports had been issued 
as of that time, but during the period from February 
through September 2012, the Committee did table 
two items with the Legislature, both requesting a 
Speaker's Warrant for the appearance of Dr. Chris 
Mazza at the hearings on our Ornge special report. 



CANADIAN COUNCIL OF PUBLIC 
ACCOUNTS COMMITTEES 

The Canadian Council of Public Accounts Commit- 
tees (CCPAC) consists of delegates from federal, 
provincial and territorial public accounts com- 
mittees from across Canada. CCPAC holds a joint 
annual conference with the Canadian Council of 
Legislative Auditors to discuss issues of mutual 
interest. 

The 33rd annual conference was hosted by 
Nunavut and was held in Iqaluit from August 19 
to 21, 2012. 



Chapter 7 



The Office of the 
Auditor General of Ontario 



General Overview 




The Office of the Auditor General of Ontario 
(Office) serves the Legislative Assembly and the 
citizens of Ontario by conducting value-for-money 
and financial audits and reviews and reporting on 
them. By doing this, the Office helps the Legislative 
Assembly hold the government, its administrators 
and grant recipients accountable for how prudently 
they spend public funds and for the value they 
obtain, on behalf of Ontario taxpayers, for the 
money spent. 

The work of the Office is performed under the 
authority of the Auditor General Act. In addition, 
under the Government Advertising Act, 2004, the 
Auditor General is responsible for reviewing and 
deciding whether or not to approve certain types of 
proposed government advertising (see Chapter 5 
for more details on the Office's advertising review 
function). Both acts can be found at www.e-laws. 
gov.on.ca. 

In an election year, the Auditor General is 
also required to review the reasonableness of the 
government's pre-election report on its expecta- 
tions for the financial performance of the province 
over the next three fiscal years. The government's 
most recent pre-election report was issued on 
April 26, 2011, and the results of our review were 
released on June 28, 2011. 



VALUE-FOR-MONEY AUDITS IN THE 
ANNUAL REPORT 

About two-thirds of the Office's work relates to 
value-for-money auditing. The Office's value-for- 
money audits are assessments of how well a given 
"auditee" (the entity that we audit) manages and 
administers its programs or activities. And while 
auditors typically focus on the numbers, value-for- 
money audits delve into the underlying operations 
to assess the level of services being delivered to the 
public and the relative cost-effectiveness of these 
services. The Office has the authority to conduct 
value-for-money audits of the following entities: 

• Ontario government ministries; 

• Crown agencies; 

• Crown-controlled corporations; and 

• organizations in the broader public sector 
that receive government grants (for example, 
hospitals, universities, community colleges, 
school boards, long-term-care homes, chil- 
dren's aid societies and agencies that provide 
mental-health services). 

The Auditor General Act (Act) identifies the 
broad criteria to be considered in this assessment: 

• Money should be spent with due regard for 
economy. 
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• Money should be spent with due regard for 
efficiency. 

• Appropriate procedures should be in place to 
measure and report on the effectiveness of 
programs. 

The Act requires that, if the Auditor General 
observes instances where the three value-for-money 
criteria have not been met, he or she report on 
them. More specific criteria that relate directly to 
the operations of the particular ministry, program 
or organization being audited are also developed 
for each value-for-money audit. 

The Act also requires that the Auditor report on 
instances where the following was observed: 

• Accounts were not properly kept or public 
money was not fully accounted for. 

• Essential records were not maintained or the 
rules and procedures applied were not suf- 
ficient to: 

• safeguard and control public property; 

• check effectively the assessment, collection 
and proper allocation of revenue; or 

• ensure that expenditures were made only 
as authorized. 

• Money was expended other than for the pur- 
poses for which it was appropriated. 

Assessing the extent to which the auditee was 
controlling against these risks is technically "com- 
pliance" audit work but is generally incorporated 
into both value-for-money audits and "attest" audits 
(discussed in a later section). Other compliance 
work that is typically included in our value-for- 
money audits is: 

• identifying the key provisions in legislation 
and the authorities that govern the auditee or 
the auditee's programs and activities as well 
as those that the auditee's management is 
responsible for administering; and 

• performing the tests and procedures we deem 
necessary to obtain reasonable assurance 
that the auditee's management has complied 
with these key legislation and authority 
requirements. 



Government programs and activities are the 
result of government policy decisions. Thus, we 
could say that our value-for-money audits focus on 
how well management is administering and execut- 
ing government policy decisions. It is important to 
note, however, that in doing so we do not comment 
on the merits of government policy. Rather, it is the 
Legislative Assembly that holds the government 
accountable for policy matters. The Legislative 
Assembly continually monitors and challenges gov- 
ernment policies through questions during legisla- 
tive sessions and through reviews of legislation and 
expenditure estimates. 

In planning, performing and reporting on our 
value-for-money work, we follow the relevant 
professional standards established by the Canadian 
Institute of Chartered Accountants. These stan- 
dards require that we have processes for ensuring 
the quality, integrity and value of our work. Some 
of the processes we use are described below. 

Selecting What to Audit 

The Office audits major ministry programs and 
activities at approximately five- to seven-year inter- 
vals. We do not audit organizations in the broader 
public sector and Crown-controlled corporations on 
the same cycle because there are such a great num- 
ber of them and their activities are so numerous 
and diverse. Since our mandate expanded in 2004 
to allow us to audit these entities, our audits have 
covered a wide range of topics in several sectors, 
including health (hospitals, long-term-care homes, 
and mental-health service providers), education 
(school boards, universities and colleges) and social 
services (Children's Aid societies and social service 
agencies), as well as several large Crown-controlled 
corporations. 

In selecting what program, activity or organiza- 
tion to audit each year, we consider how great the 
risk is that an auditee is not meeting the three legis- 
lated value-for-money criteria. To help us choose 
higher-risk audits, we consider factors such as: 
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• the results of previous audits and related 
follow-ups; 

• the total revenues or expenditures involved; 

• the impact of the program, activity or organ- 
ization on the public; 

• the complexity and diversity of the auditee's 
operations; 

• recent significant changes in the auditee's 
operations; and 

• the significance of the issues an audit might 
identify. 

We also consider whether the benefits of con- 
ducting the audit justify its costs. 

Another factor we take into account in the selec- 
tion process is what work the auditee's internal 
auditors have completed or planned. Depending 
on that work, we may defer an audit or change our 
audit's scope to avoid duplication of effort. In other 
cases, we do not diminish the scope of our audit 
but rely on and present the results of internal audit 
work in our audit report. 

Setting Audit Objectives, Audit Criteria and 
Assurance Levels 

When we begin an audit, we set an objective for 
what we want to achieve. We then develop suitable 
audit criteria that cover the key systems, policies 
and procedures that should be in place and operat- 
ing effectively. Developing criteria involves exten- 
sively researching sources such as recognized bodies 
of experts; other bodies or jurisdictions delivering 
similar programs and services; management's own 
policies and procedures; applicable criteria success- 
fully applied in other audits or reviews; and applic- 
able laws, regulations and other authorities. 

To further ensure their suitability, the criteria 
we develop are discussed with the senior manage- 
ment responsible for the program or activity at the 
planning stage of the audit. 

The next step is designing and conducting tests 
and procedures to address our audit objective 
and criteria, so that we can reach a conclusion 
regarding our audit objective and make observa- 



tions and recommendations. Each audit report has 
a section entitled "Audit Objective and Scope," in 
which the audit objective is stated and the scope of 
our work is explained. 

Conducting tests and procedures to gather 
information has its limitations. We therefore cannot 
provide what is called an "absolute level of assur- 
ance" that our audit work identifies all significant 
matters. Other factors also contribute to this. For 
example, we may conclude that the auditee had a 
control system in place for a process or procedure 
that was working effectively to prevent a particular 
problem from occurring — but auditee management 
or staff might be able to circumvent such control 
systems, so we cannot guarantee that the prob- 
lem will never arise. Also, much of the evidence 
available for concluding on our objective is more 
persuasive than it is conclusive, and we must rely 
on professional judgment in much of our work — for 
example, in interpreting information. 

For all these reasons, the assurance that we plan 
for our work to provide is at an "audit level" — the 
highest reasonable level of assurance that we 
can obtain using our regular audit procedures. 
Specifically, an audit level of assurance is obtained 
by interviewing management and analyzing the 
information it provides; examining and testing sys- 
tems, procedures and transactions; confirming facts 
with independent sources; and, where necessary 
because we are examining a highly technical area, 
obtaining expert assistance and advice. 

With respect to the information that manage- 
ment provides, under the Act we are entitled to 
have access to all relevant information and records 
necessary to the performance of our duties. Out of 
respect for the principle of Cabinet privilege, we 
do not seek access to the deliberations of Cabinet. 
However, the Office can access virtually all other 
information contained in Cabinet submissions or 
decisions that we deem necessary to fulfill our 
responsibilities under the Act. 

Infrequently, the Office will perform a review 
rather than an audit. A review provides a moder- 
ate level of assurance, obtained primarily through 
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inquiries and discussions with management; analy- 
ses of information management provides; and only 
limited examination and testing of systems, proced- 
ures and transactions. We perform reviews when, 
for example, providing a higher level of assurance 
has prohibitive costs or is unnecessary, the Auditor 
General Act does not allow for a certain program 
or activity to be audited or other factors relating 
to the nature of the program or activity make a 
review more appropriate than an audit. In the 2011 
audit year, we conducted a review of the electri- 
city sector's stranded debt, which complemented 
our related value-for-money audits of renewable 
energy initiatives and regulatory oversight of the 
electricity sector. In 2009, we conducted a review 
of the unfunded liability of the Workplace Safety 
and Insurance Board that was well received by the 
Standing Committee on Public Accounts, and the 
Committee has shown an ongoing interest in the 
actions being taken to reduce that liability. This 
year, we reviewed the process used to review and 
approve the province's annual expenditure Esti- 
mates and researched other jurisdictions' estimates 
oversight practices. 

Communicating with Management 

To help ensure the factual accuracy of our observa- 
tions and conclusions, staff from our Office com- 
municate with the auditee's senior management 
throughout the value-for-money audit or review. 
Early in the process, our staff meet with manage- 
ment to discuss the objective and criteria and the 
focus of our work in general terms. During the 
audit or review, our staff meet with management 
to review progress and ensure open lines of com- 
munication. At the conclusion of on-site work, 
management is briefed on the preliminary results 
of the work. A draft report is then prepared and 
discussed with the auditee's senior management. 
The auditee's management provides written 
responses to our recommendations, and these are 
discussed and incorporated into the draft report. 
The Auditor General finalizes the draft report (on 



which the Chapter 3 section of the Annual Report 
will be based) with the deputy minister or head of 
the agency, corporation or grant-recipient organ- 
ization, after which the report is published in the 
Annual Report. 

SPECIAL REPORTS 

As required by the Act, the Office reports on its aud- 
its in an Annual Report to the Legislative Assembly. 
In addition, the Office may make a special report to 
the Legislative Assembly at any time, on any matter 
that, in the opinion of the Auditor General, should 
not be deferred until the Annual Report. 

Two sections of the Act authorize the Auditor 
General to undertake additional special work. 
Under section 16, the Standing Committee on Pub- 
lic Accounts may resolve that the Auditor General 
must examine and report on any matter respecting 
the Public Accounts. Under section 17, the Legisla- 
tive Assembly, the Standing Committee on Public 
Accounts or a minister of the Crown may require 
that the Auditor General undertake a special assign- 
ment. However, these special assignments are not 
to take precedence over the Auditor General's other 
duties. In addition, if a minister requests such an 
assignment, the Auditor General can decline or 
delay conducting the work if he or she believes it 
conflicts with other duties. 

In recent years, when we have received a special 
request under section 16 or 17, our normal practice 
has been to obtain the requester's agreement that 
the special report will be tabled in the Legislature 
on completion and made public at that time. 

In 2011, one of the audits that we had initiated 
using our risk-based selection criteria — Ornge Air 
Ambulance and Related Services — was delayed 
due to challenges in finalizing the report with the 
auditee. The Minister requested that on completion 
we table the report in the Legislative Assembly as a 
special report, which we did in March 2012. 



ATTEST AUDITS 

Attest audits are examinations of an auditee's 
financial statements. In such audits, the auditor 
expresses his or her opinion on whether the finan- 
cial statements present information on the auditee's 
operations and financial position in a way that 
is fair and that complies with certain accounting 
policies (in most cases, with Canadian generally 
accepted accounting principles) . As mentioned in 
the overview of value-for-money audits, compliance 
audit work is often incorporated into attest audit 
work. Specifically, we assess the controls for man- 
aging risks relating to improperly kept accounts; 
unaccounted-for public money; lack of record- 
keeping; inadequate safeguarding of public prop- 
erty; deficient procedures for assessing, collecting 
and properly allocating revenue; unauthorized 
expenditures; and not spending money on what it is 
intended for. 

TheAuditees 

Every year, we audit the financial statements of 
the province and the accounts of many agencies 
of the Crown. Specifically, the Act [in subsections 
9(1), (2), and (3)] requires that: 

• the Auditor General audit the accounts and 
records of the receipt and disbursement of 
public money forming part of the province's 
Consolidated Revenue Fund, whether held in 
trust or otherwise; 

• the Auditor General audit the financial state- 
ments of those agencies of the Crown that are 
not audited by another auditor; 

• public accounting firms that are appointed 
to be the auditors of certain agencies of the 
Crown perform their audits under the direc- 
tion of the Auditor General and report their 
results to the Auditor General; and 

• public accounting firms auditing Crown- 
controlled corporations deliver to the Auditor 
General a copy of the audited financial state- 
ments of the corporation and a copy of the 
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accounting firm's report of its findings and 
recommendations to management (typically 
contained in a management letter) . 

Chapter 2 discusses this year's attest audit of the 
province's consolidated financial statements. 

We do not discuss the results of attest audits of 
agencies and Crown-controlled corporations in this 
report. Agency legislation normally stipulates that 
the Auditor General's reporting responsibilities are 
to the agency's board and the minister (s) respon- 
sible for the agency. Our Office also provides copies 
of our independent auditor's reports and of the 
related agency financial statements to the deputy 
minister of the associated ministry, as well as to the 
Secretary of the Treasury Board. 

Where an agency attest audit notes areas where 
management must make improvements, the auditor 
prepares a draft findings report and discusses it 
with senior management. The report is revised 
to reflect the results of that discussion. After the 
draft report is cleared and the agency's senior 
management responds to it in writing, the auditor 
prepares a final report, which is discussed with the 
agency's audit committee if one exists. If a matter 
is so significant that we feel it should be brought to 
the attention of the Legislature, we include it in our 
Annual Report. 

Exhibit I, Part 1 lists the agencies that were 
audited during the 2011/12 audit year. The Office 
currently contracts with public accounting firms 
to audit a number of these agencies on the Office's 
behalf. Exhibit 1, Part 2 and Exhibit 2, respectively, 
list the agencies of the Crown and the Crown-con- 
trolled corporations that public accounting firms 
audited directly during the 2011/12 audit year. 

OTHER STIPULATIONS OF THE AUDITOR 
GENERAL ACT 

The Auditor General Act came about with the pas- 
sage, on November 22, 2004, of Bill 18, the Audit 
Statute Law Amendment Act, which received Royal 
Assent on November 30, 2004. The purpose of 
Bill 18 was to make certain amendments to the 
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Audit Act to enhance the ability of the Office to 
serve the Legislative Assembly. The most significant 
amendment contained in Bill 18 was the expansion 
of the Office's value-for-money audit mandate to 
organizations in the broader public sector that 
receive government grants. This 2012 Annual 
Report marks the seventh year of our expanded 
audit mandate. 

Appointment of Auditor General 

Under the Act, the Auditor General is appointed as 
an officer of the Legislative Assembly by the Lieu- 
tenant Governor in Council — that is, the Lieutenant 
Governor appoints the Auditor General on the 
advice of the Executive Council (the Cabinet) . The 
appointment is made "on the address of the Assem- 
bly," meaning that the appointee must be approved 
by the Legislative Assembly. The Act also requires 
that the Chair of the Standing Committee on Public 
Accounts — who, under the Standing Orders of the 
Legislative Assembly, is a member of the official 
opposition — be consulted before the appointment is 
made (for more information on the Committee, see 
Chapter 6). In the selection of the last two Auditors 
General, an open competitive selection process was 
conducted, with representatives from the three 
major political parties sitting on the selection com- 
mittee. The committee then made its recommenda- 
tion to the Legislative Assembly. 

Independence 

The Auditor General and staff of the Office are 
independent of the government and its administra- 
tion. This independence is an essential safeguard 
that enables the Office to fulfill its auditing and 
reporting responsibilities objectively and fairly. 

The Auditor General is appointed to a 10-year, 
non-renewable term, and can be dismissed only for 
cause by the Legislative Assembly. Consequently, 
the Auditor General maintains an arm's-length dis- 
tance from the government and the political parties 
in the Legislative Assembly and is thus free to fulfill 



the Office's legislated mandate without political 
pressure. 

The Board of Internal Economy — an all-party 
legislative committee that is independent of the 
government's administrative process — reviews and 
approves the Office's budget, which is subsequently 
laid before the Legislative Assembly. As required 
by the Act, the Office's expenditures relating to the 
2011/12 fiscal year have been audited by a firm of 
chartered accountants, and the audited financial 
statements of the Office are submitted to the Board 
and subsequently must be tabled in the Legislative 
Assembly. The audited statements and related dis- 
cussion of expenditures for the year are presented 
at the end of this chapter. 

CONFIDENTIALITY OF WORKING PAPERS 

In the course of our reporting activities, we prepare 
draft audit reports and findings reports that are 
considered to be an integral part of our audit work- 
ing papers. It should be noted that these working 
papers, according to section 19 of the Auditor Gen- 
eral Act, do not have to be laid before the Legislative 
Assembly or any of its committees. As well, because 
our Office is exempt from the Freedom of Informa- 
tion and Protection of Privacy Act, our draft reports 
and audit working papers, which include all infor- 
mation obtained during the course of an audit from 
the auditee, cannot be accessed from our Office, 
thus further ensuring confidentiality. 

CODE OF PROFESSIONAL CONDUCT 

The Office has a Code of Professional Conduct to 
encourage staff to maintain high professional stan- 
dards and ensure a professional work environment. 
The Code is intended to be a general statement of 
philosophy, principles and rules regarding conduct 
for employees of the Office, who have a duty to 
conduct themselves in a professional manner and to 
strive to achieve the highest standards of behaviour, 
competence and integrity in their work. 
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The Code explains why these expectations exist 
and further describes the Office's responsibilities to 
the Legislative Assembly, the public and our aud- 
itees. The Code also provides guidance on disclo- 
sure requirements and the steps to be taken to avoid 
conflict-of-interest situations. All employees are 
required to complete an annual conflict-of-interest 
declaration and undergo a police security check on 
hiring and every five years thereafter. 



Office Organization and 
Personnel 



The Office is organized into portfolio teams — a 
framework that attempts to align related audit 
entities and to foster expertise in the various areas 
of audit activity. The portfolios, which are loosely 
based on the government's own ministry organiza- 
tion, are each headed by a Director, who oversees 
and is responsible for the audits within the assigned 
portfolio. Assisting the Directors and rounding out 
the teams are a number of audit Managers and vari- 
ous other audit staff (see Figure 1). 

The Auditor General, the Deputy Auditor Gen- 
eral, the Directors, and the Managers of Human 
Resources and of Communications and Govern- 
ment Advertising Review make up the Office's 
Senior Management Committee. 



Canadian Council of 
Legislative Auditors 




This year, Nunavut hosted the 40th annual meeting 
of the Canadian Council of Legislative Auditors 
(CCOLA) in Iqaluit, from August 19 to 21, 2012. 
This annual gathering has, for a number of years, 
been held jointly with the annual conference of the 
Canadian Council of Public Accounts Committees. 
It brings together legislative auditors and members 



of the Standing Committees on Public Accounts 
from the federal government and the provinces and 
territories, and provides a useful forum for sharing 
ideas and exchanging information. 




International Visitors 



As an acknowledged leader in value-for-money 
auditing, the Office periodically receives requests 
to meet with visitors and delegations from abroad 
to discuss the roles and responsibilities of the Office 
and to share our value-for-money and other audit 
experiences with them. During the audit year cov- 
ered by this report, the Office met with several dele- 
gations from other countries. In the last few years, 
delegations from China have been the most frequent 
visitors. The Office also participated in a staff 
exchange with the Auditor General of Tasmania. 



Results Produced by the 
Office This Year 



The 2011/12 fiscal year was another successful year 
for the Office. In total, we conducted 12 value-for- 
money audits, issued a special report under section 
17 on Ornge Air Ambulance and Related Services, 
and reviewed the Legislature's process for reviewing 
the province's annual Expenditure Estimates. This 
year our value-for-money audits in the broader 
public sector examined aspects of a number of 
provincially funded agencies such as Cancer Care 
Ontario's cancer screening programs, Metrolinx's 
rapid transit plan implementation, services provided 
by independent health facilities, the long-term-care 
home placement process, youth justice service pro- 
viders, and teaching quality at universities. In addi- 
tion, we examined a variety of government activities 
also of importance to Ontarians, including educa- 
tion of Aboriginal students, the Ontario Diabetes 
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Figure 1: Office Organization, September 30, 2012 



Auditor General 



Jim McCarter 



Deputy Auditor General 



Gary Peall 



Human Resources 



Annemarie Wiebe, Manager 
Shayna Whiteford 



Quality Assurance and Special Projects 



Paul Amodeo, Manager 
Kristin Snowden, Manager 
Shariq Saeed 



Standards and Research 



Rebecca Yosipovich, Manager 



Operations 



Administration 

Maureen Bissonnette 
Sohani Myers 
Shanta Persaud 
Christine Wu 

Communications and Government Advertising Review 

Christine Pedias, Manager 
Mariana Green 
Shirley McGibbon 
Tiina Randoja 
Information Technology 
Shams AH 
Peter Lee 



Financial Statement Audit Portfolios and Staff* 



Crown Agencies (1), Finance 



Crown Agencies (2) 



John McDowell, Director 
Walter Allan, Manager 
Tom Chatzidimos 
Kandy Fletcher 
Mary Romano 
Megan Sim 



Laura Bell, Director 
Teresa Carello, Manager 
Izabela Beben 
Margaret Chen 
Constantino De Sousa 



Roger Munroe 
Zachary Thomas 
Cynthia Tso 



Public Accounts 



Bill Pelow, Director 
Allen Fung 

Georgegiana Tanudjaja 



Value-for-money Audit Porfolios and Staff* 



Community and Social Services, 
and Revenue 



Vanna Gotsis, Director 
Sandy Chan, Manager 
Tino Bove 
Inna Guelfand 
Michael Okulicz 



Zhenya Stekovic 
Janet Wan 



Education and Training 



Gerard Fitzmaurice, Director 
Emanuel Tsikritsis, Manager 



Tara Beheshti 
Johan Boer 
Zahra Jaffer 
Rumi Janmohamed 
Michael Katsevman 



Nina Khant 
Mythili Pratheeskar 
Mark Smith 
Ellen Tepelenas 
Dora Ulisse 



Environment and Natural Resources 



Gus Chagani, Director 

Kim Cho, Manager 

Nick Stavropoulos, Manager 



Bartosz Amerski 
Marcia DeSouza 
Katrina Exaltacion 
Lauren Hanna 
Kristy Ho 



Li-Lian Koh 
Shreya Shah 
Alexander Truong 
Jing Wang 



Health 



Rudolph Chiu, Director 

Gigi Yip, Manager 

Denise Young, Manager 

Ariane Chan Lisa Li 

Anita Cheung Oscar Rodriguez 

Helen Chow Alia Volodina 



Health and Long-term-care Providers 



Susan Klein, Director 

Wendy Cumbo, Manager 

Naomi Herberg, Manager 

Kevin Aro Ingrid Goh 

Sally Chang Veronica Ho 

Dimitar Dimitrov Linde Qiu 

Jennifer Fung Tiffany Yau 



Justice and Regulatory 



Vince Mazzone, Director 

Rick MacNeil, Manager 

Fraser Rogers, Manager 

Vivian Sin, Manager 

RashmeetGill Alice Nowak 

Kiran Grewal Ruchir Patel 

Tanmay Gupta Brian Wanchuk 

Alfred Kiang Robyn Wilson 

Margaret Lam Michael Yarmolinsky 

Wendy Ng Celia Yeung 



*Staff below manager level shift between portfolios to address seasonal financial statement audit workload pressures. 
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Strategy, the Drive Clean Program, the collection 
of overdue taxes, the Criminal Law Division of the 
Ministry of the Attorney General, and aspects of the 
Ontario Provincial Police services. 

As mentioned in the earlier Attest Audits sec- 
tion, we are responsible for auditing the province's 
consolidated financial statements (further dis- 
cussed in Chapter 2), as well as the statements of 
more than 40 Crown agencies. We again met all of 
our key financial-statement audit deadlines while 
continuing our investment in training to success- 
fully implement significant revisions to accounting 
and assurance standards and methodology for 
conducting our financial-statement audits. 

We successfully met our review responsibilities 
under the Government Advertising Act, 2004, as 
further discussed in Chapter 5. 

The results produced by the Office this year 
would clearly not have been possible without the 
hard work and dedication of our staff, as well as 
that of our agent auditors, contract staff and expert 
advisers. 



Financial Accountability 



The following discussion and our financial state- 
ments reproduced at the end of this chapter outline 



the Office's financial results for the 2011/12 fiscal 
year. This year our financial statements have been 
prepared in accordance with public-sector account- 
ing standards. In accordance with these standards 
we have presented for the first time a breakdown of 
our expenses by the four main activities our Office 
is responsible for: value-for-money audits, financial 
statement audits, our review of the government's 
2011 pre-election report (as described earlier in 
this chapter, the Auditor General is required to 
review the reasonableness of the government's 
pre-election report in an election year, which 
2011 was) and our government advertising review 
responsibilities. This breakdown is provided in note 
10 to the financial statements and indicates that 
almost two-thirds of our resources were used to 
perform value-for-money audits, the activity that is 
of most interest to the Standing Committee on Pub- 
lic Accounts. About 30% was devoted to completing 
the audits of the annual financial statements of the 
province and some 40 of its agencies. The remain- 
ing 3% was devoted to the other two statutory 
responsibilities. These percentages changed only 
slightly from 2011, mostly because we diverted 
some value-for-money resources to help complete 
the review of the pre-election report early in the 
2011/12 fiscal year. 

Figure 2 provides a comparison of our approved 
budget and expenditures over the last five years. 



Figure 2: Five-year Comparison of Spending (Accrual Basis) ($ 000) 

Prepared by the Office of trie Auditor General of Ontario 





2007/08 


2008/09 


2009/10 


2010/11 


2011/12 


Approved budget 


15,308 


16,245 


16,224 


16,224 


16,224 


Actual expenses 


Salaries and benefits 


9,999 


10,279 


10,862 


11,228 


11,039 


Professional and other services 


1,525 


1,776 


1,489 


1,491 


1,667 


Rent 


1,048 


1,051 


1,069 


1,036 


1,016 


Travel and communications 


397 


332 


360 


337 


303 


Other 


1,033 


1,096 


1,073 


1,071 


1,216 


Total 


14,002 


14,534 


14,853 


15,163 


15,241 


Returned to province* 


1,608 


1,561 


1,498 


1,222 


997 



"These amounts are typically slightly different than the excess of appropriation over expenses as a result of non-cash expenses (such as amortization of capital 
assets, deferred lease inducements and employee future benefit accruals). 
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Figure 3 presents the major components of our 
spending and shows that more than 72% (74% in 
2010/11) related to salary and benefit costs for our 
staff, while professional and other services and rent 
comprised most of the remainder. These propor- 
tions have been relatively stable in recent years. 

Overall, our expenses increased just 0.5% 
(2.1% in 2010/11) and were again significantly 
under budget. Our budget has been frozen over the 
last four years, yet we have successfully delivered 
on our Office's mandate while returning unspent 
funds totalling $5.3 million. The main reason for 
our ability to come in under budget is that we have 
historically faced challenges in hiring and retain- 
ing qualified professional staff in the competitive 
Toronto job market — our public-service salary 
ranges have simply not kept pace with compensa- 
tion increases for such professionals in the private 
sector. In addition, we have been reluctant to fully 
staff up given the tight fiscal circumstances the 
province faces. 

A more detailed discussion of the changes in our 
expenses and some of the challenges we are facing 
follows. 

SALARIES AND BENEFITS 

Our salary costs declined 3.2% this year while 
benefit costs rose 5.2% from the previous year. 

With the legislated freeze on salary ranges 
again this year, any increases due to promotions 
earned by trainees who obtained their professional 
accounting designations during the year and for 
those staff demonstrating the ability to take on 
additional responsibilities were more than offset 
by delays in replacing retiring and departing staff. 
As well, parental leaves and severance payments 
to retirees this year, while lowering salary costs, 
increased both our benefit costs and our need for 
contract professional services to fill in. We continue 
to employ fewer student trainees, as many of our 
trainees earned their professional accounting desig- 
nation during the year and remained with us. To be 
competitive, we must pay our newly qualified staff 



Figure 3: Spending by Major Expenditure Category, 
2011/12 

Prepared by trie Office of trie Auditor General of Ontario 



Other (8.0%) 
Rent (6.7%) 



Travel and 

communications (2.0%) 



Professional and 
other services 
(10.9%) 



Salaries and 
benefits (72.4%) 




considerably more than they were paid as trainees. 
Salaries for qualified accountants rise fairly quickly 
in the first five years following qualification. 

With the economic uncertainty and the con- 
tinuing need for cost containment, we remained 
cautious about staffing up when staff departed, 
delaying the replacement of retiring senior staff 
and hiring experienced but more junior staff as 
opportunities arose. As a result, our average staff- 
ing over the course of this year declined to about 
102, down from 104 in the previous year. Staff 
departures continue as the market for professional 
accountants has remained fairly robust despite 
economic uncertainties. Our hiring continues 
to be primarily at more junior levels, given that 
our salaries and benefits are competitive at these 
levels. We quickly fall behind private- and broader- 
public-sector salary scales for more experienced 
professional accountants. This is one reason that, as 
Figure 4 shows, we still have a number of unfilled 
positions. The growing complexity of our audits 
demands that we use highly qualified, experienced 
staff as much as possible. The challenge of main- 
taining and enhancing our capacity to perform 
these audits will only increase as more of our most 
experienced staff retire over the next few years. 

Under the Act, our salary levels must be compar- 
able to the salary ranges of similar positions in the 
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Figure 4: Staffing, 2005/06-2011/12 

Prepared by the Office of trie Auditor General of Ontario 

i i Unfilled positions 
i i Actual average FTEs 
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government. These ranges remain uncompetitive 
with the salaries that both the not-for-profit and the 
private sectors offer. According to the most recent 
survey by the Canadian Institute of Chartered 
Accountants published in 2011, average salaries 
for CAs in government ($111,200) were 14% lower 
than those in the not-for-profit sector ($129,900) 
and, most importantly, 26% lower than those work- 
ing for professional service CA firms ($150,400), 
which are our primary competitors for professional 
accountants. The salaries of our highest-paid staff 
in the 2011 calendar year are disclosed in Note 8 to 
our financial statements. 

PROFESSIONAL AND OTHER SERVICES 

These services include both contract professionals 
and contract CA firms and represent our next most 
significant area of expenditure at almost 11% of 
total expenditures. Such costs were higher than 
last year due to an increase in parental leaves 
requiring contract fill-ins as well as the purchase of 
IT consulting expertise to enhance the Office's IT 
infrastructure. 

We continue to have to rely on contract profes- 
sionals to meet our legislated responsibilities 



given more complex work and tight deadlines for 
finalizing the financial-statement audits of Crown 
agencies and the province. We also believe that 
using more contract staff to fill temporary needs, 
such as for parental leaves, is a prudent approach 
to staffing, particularly during uncertain economic 
times, in that it provides more flexibility and less 
disruption if significant in-year cuts to our budget 
are requested. Also, even during the economic 
downturn it has remained difficult for us to reach 
our approved full complement given our uncom- 
petitive salary levels, particularly for professionals 
with several years of post-qualifying experience. 
Further, after four years of budget freezes we can 
no longer realistically afford to staff up to our 
approved complement of 117 staff. 

We continue to incur higher contract costs for 
CA firms we work with because of the higher salar- 
ies they pay their staff and the additional hours 
required to implement ongoing changes to account- 
ing and assurance standards. We continue to test 
the market for such services as contracts expire. 

RENT 

Our costs for accommodation were again slightly 
less than last year, owing primarily to a decline 
in building operating costs, particularly utilities. 
Accommodation costs declined as a percentage of 
total spending and will decline further because the 
Office aggressively negotiated a rent reduction as 
part of the lease renewal terms commencing in the 
fall of 2011. 

TRAVEL AND COMMUNICATIONS 

Our travel and communications costs declined 
more than 10% from last year. The value-for- 
money audits we selected generally required less 
travel than usual. Last year, three audits required 
extensive travel: Forest Management Program, Sup- 
portive Services for People with Disabilities, and 
Student Success Initiatives. In general, however, 
we have been incurring significantly more travel 
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costs since the expansion of our mandate to audit 
broader-public-sector organizations. 



OTHER 

Other costs include asset amortization, supplies 
and equipment maintenance, training and statutory 
expenses. Such costs were 13.5% higher than last 
year, primarily due to contract staff and special- 
ist expertise we hired to complete our statutory 
review of the 2011 Pre-Election Report on Ontario's 
Finances, which we reported on in June 2011. We 
also spent $70,000 more on supplies and equip- 
ment, primarily for additional IT support licences 
and audit software licences as we phase in the 
implementation of electronic working papers. 
Training expenses increased $35,000 to help our 
staff and the agencies we audit prepare for the 
transition to new financial reporting frameworks. 
These increases were partially offset by reductions 
in expert advisory services required to administer 
the Government Advertising Act, 2004, as the 
volume of ads requiring their review declined this 
year. Smaller changes in expenses for the Auditor 
General's salary, the transfer payment to CCAF and 
equipment amortization were largely offsetting. 



The Office of the Auditor General of Ontario 



FINANCIAL STATEMENTS 




Office of the Auditor General of Ontario 
Bureau du verificateur general de I'Ontario 



MANAGEMENT'S RESPONSIBILITY FOR FINANCIAL STATEMENTS 



The accompanying financial statements of the Office of the Auditor General are the 
responsibility of management of the Office. Management has prepared the financial 
statements to comply with the Auditor General Act and with Canadian public sector 
accounting principles. 

Management maintains a system of internal controls that provides reasonable assurance that 
transactions are appropriately authorized, assets are adequately safeguarded, appropriations 
are not exceeded, and the financial information contained in these financial statements is 
reliable and accurate. 

The financial statements have been audited by the firm of Adams & Miles LLP, Chartered 
Accountants. Their report to the Board of Internal Economy, stating the scope of their 
examination and opinion on the financial statements, appears on the following page. 



Jim McCarter, FCA 
Auditor General 
August 17,2012 



Gary R. Peall, CA 
Deputy Auditor General 
August 17,2012 
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www.auditor.on.ca 



2012 Annual Report of the Office of the Auditor General of Ontario 



ADAMS & MILES LLP 

Chartered Accountants 



501-2550 Victoria Park Ave. 
Toronto, ON M2J 5A9 
Tel 416 502.2201 
Fax 416 502.2210 



200-195 County Court Blvd. 
Brampton, ON L6W 4P7 
Tel 905 459.5605 
Fax 905 459.2893 



INDEPENDENT AUDITORS' REPORT 

To the Board of Internal Economy of 
Legislative Assembly of Ontario 

We have audited the accompanying financial statements of the Office of the Auditor General of 
Ontario, which comprise the statement of financial position as at March 31 , 2012, March 31 , 201 1 , 
and April 1 , 2010 and the statements of operations and accumulated deficit and cash flows for the 
years ended March 31, 2012 and March 31, 2011, and a summary of significant accounting 
policies and other explanatory information. 

Management's Responsibility for the Financial Statements 

Management is responsible for the preparation and fair presentation of these financial statements 
in accordance with Canadian public sector accounting standards, and for such internal control as 
management determines is necessary to enable the preparation of financial statements that are 
free from material misstatement, whether due to fraud or error. 

Auditors' Responsibility 

Our responsibility is to express an opinion on these financial statements based on our audit. We 
conducted our audit in accordance with Canadian generally accepted auditing standards. Those 
standards require that we comply with ethical requirements and plan and perform the audit to 
obtain reasonable assurance about whether the financial statements are free of material 
misstatement. 

An audit involves performing procedures to obtain audit evidence about the amounts and 
disclosures in the financial statements. The procedures selected depend on our judgment, 
including the assessment of the risks of material misstatement of the financial statements, whether 
due to fraud or error. In making those risk assessments, we consider internal control relevant to 
the preparation and fair presentation of the financial statements in order to design audit 
procedures that are appropriate in the circumstances, but not for the purpose of expressing an 
opinion on the effectiveness of the Office of the Auditor General of Ontario's internal control. An 
audit also includes evaluating the appropriateness of accounting policies used and the 
reasonableness of accounting estimates made by management, as well as evaluating the overall 
presentation of the financial statements. 

We believe that the audit evidence we have obtained is sufficient and appropriate to provide a 
basis for our audit opinion. 



In our opinion, the financial statements present fairly, in all material respects, the financial position 
of the Office of the Auditor General of Ontario as at March 31 , 201 2, March 31 , 201 1 , and April 1 , 
201 0, and the results of its operations and its cash flows for the years ended March 31,2012 and 
March 31 , 201 1 in accordance with Canadian public sector accounting standards. 



Opinion 



Adams d£ Miles LL<P 



Chartered Accountants 
Licensed Public Accountants 



Toronto, Canada 
August 17, 2012 



www.adamsmifes.com 



An independent firm associated 
with AGN International Ltd. 



The Office of the Auditor General of Ontario 



Office of the Auditor General of Ontario 

Statement of Financial Position 
As at March 31, 2012 



March 31, 2012 March 31, 2011 April 1,2010 
(Note 12) (Note 2) 

$ $ $ 



Financial assets 



Cash and cash equivalents 
Harmonized sales taxes recoverable 
Due from Consolidated Revenue Fund 
Lease inducement receivable (Note 11) 


290,695 
122,604 
237,316 
322,225 


500,170 
128,927 
49,194 


370,802 
754,098 




972,840 


678,291 


1,124,900 


Liabilities 

Accounts payable and accrued liabilities (Note 2) 

Deferred lease inducement (Note 11) 

Accrued employee benefits obligation [Note 6(B)] 


1,647,041 
308,799 
2,172,000 


1,739,291 
1,988,000 


2,129,900 
1,922,000 




4,127,840 


3,727,291 


4,051,900 


Net financial debt 


(3,155,000) 


(3,049,000) 


(2,927,000) 


Non-financial assets 

Tangible capital assets (Note 4) 


593,979 


501,904 


540,543 


Accumulated deficit (Note 2) 


(2,561,021) 


(2,547,096) 


(2,386,457) 



Commitments (Note 7) 
Measurement uncertainty (Note 3(E)) 

See accompanying notes to financial statements. 



Approved by the Office of the Auditor General of Ontario: 



Jim McCarter, FCA 
Auditor General 



Gary Peall, CA 
Deputy Auditor General 
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Office of the Auditor General of Ontario 

Statement of Operations and Accumulated Deficit 
For the Year Ended March 31, 2012 and March 31, 2011 



2012 2012 2011 

Budget Actual Actual 

(Unaudited -Note 13) (Note 12) 

$ $ $ 

Expenses 



Salaries and wages 


9,755,400 


8,953,561 


9,245,160 


Employee benefits (Notes 2 & 6) 


2,041,200 


2,085,050 


1,982,895 


Office rent 


1,062,400 


1,016,280 


1,035,522 


Professional and other services 


1,714,500 


1,666,589 


1,490,944 


Amortization of capital assets 




324,489 


339,316 


Travel and communication 


418,800 


303,072 


337,301 


Training and development 


378,600 


165,152 


130,700 


Supplies and equipment 


377,500 


208,311 


136,574 


Transfer payment: CCAF-FCVI Inc. 


73,000 


72,989 


61,775 


Statutory expenses: Auditor General Act 


222,700 


246,575 


229,147 


Government Advertising Act 


30,000 


10,942 


65,060 


Statutory services 


150,000 


187,582 


108,434 


Total expenses (Notes 9 & 10) 


16,224,100 


15,240,592 


15,162,828 



Revenue 

Consolidated Revenue Fund - Voted appropriation [Note 3(B)] 16,224,100 16,224,100 16,224,100 

Excess of appropriation over expenses 983,508 1 ,061 ,272 

Less: returned to the Province [Note 3(B)] 997,433 1 ,221 ,91 1 



Net operations deficiency 

Accumulated deficit, as restated, beginning of year (Note 2) 
Accumulated deficit, end of year 



(13,925) 


(160,639) 


(2,547,096) 


(2,386,457) 


(2,561,021) 


(2,547,096) 



See accompanying notes to financial statements. 
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Office of the Auditor General of Ontario 

Statement of Cash Flows 

For the Year Ended March 31, 2012 and March 31, 2011 





2012 


2011 




$ 


$ 


Net inflow (outflow) of cash related to the following activities: 






Operating transactions 






Net operations deficiency 


(13,925) 


(160,639) 


Amortization of capital assets 


324,489 


339,316 


Accrued employee benefits obligation 


184,000 


66,000 




494,564 


244,677 


Changes in non-cash working capital 






Decrease (increase) in due from Consolidated Revenue Fund 


(188,122) 


704,904 


increase in lease muucemeni receivaDie 


/Q99 OOR\ 




Decrease (increase) in harmonized sales taxes recoverable 


6,323 


(128,927) 


Decrease in accounts payable and accrued liabilities 


(92,250) 


(390,609) 


Increase in deferred lease inducement 


308,799 






(287,475) 


185,368 


Cash provided by operating transactions 


207,089 


430,045 


Capital transactions 






Purchase of tangible capital assets 


(416,564) 


(300,677) 


Investing transactions 






Financing transactions 






Increase (decrease) in cash and cash equivalents 


(209,475) 


129,368 


Cash and cash equivalents, beginning of year 


500,170 


370,802 


Cash and cash equivalents, end of year 


290,695 


500,170 



See accompanying notes to financial statements. 
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Office of the Auditor General of Ontario 

Notes to Financial Statements 

For the Year Ended March 31, 2012 and March 31, 2011 

1. Nature of Operations 

In accordance with the provisions of the Auditor General Act and various other statutes and authorities, the 
Auditor General conducts independent audits of government programs, of institutions in the broader public sector 
that receive government grants, and of the fairness of the financial statements of the Province and numerous 
agencies of the Crown. In doing so, the Office of the Auditor General promotes accountability and value-for- 
money in government operations and in broader public sector organizations. 

Additionally, under the Government Advertising Act, 2004, the Auditor General is required to review specified 
types of advertising, printed matter or reviewable messages proposed by government offices to determine 
whether they meet the standards required by the Act. 

Under both Acts, the Auditor General reports directly to the Legislative Assembly. 

As required by the Fiscal Transparency and Accountability Act, 2004, in June 201 1 the Auditor General also 
reported on the 201 1 Pre-Election Report prepared by the Ministry of Finance. 

2. Conversion to Public Sector Accounting Standards 

Commencing with the 2012 fiscal year, the Office of the Auditor General of Ontario ("Office") has adopted 
Canadian public sector accounting ("PSA") standards. These financial statements are the first financial 
statements for which the Office has applied Canadian public sector accounting standards. 

One adjustment to previously reported figures resulted from the adoption of these accounting standards. 
Previously, the Office did not record an obligation for future employee benefits related to compensated absences 
under the Management Compensation Option ("MCO"). PSA standards require that a liability and an expense be 
recognized for post-employment benefits and compensated absences earned in the period in which employees 
render services to the Office in return for the benefits. An adjustment was made to recognize a liability and an 
expense related to accumulated unused MCO credits and has been applied retroactively with restatement of prior 
periods. 

The resulting increase to accounts payable and accrued liabilities at April 1, 2010 was $209,000. Employee 
benefits expense decreased by $5,000 for fiscal 2010/2011 and accounts payable and accrued liabilities recorded 
at March 31, 2011 decreased to $204,000. The impact of the conversion to PSA standards on the accumulated 
deficit at March 31, 2011 and the comparative accumulated deficit is presented below. 

2011 2010 
$ $_ 

Accumulated deficit as originally reported, beginning of year (2,343,096) (2,177,457) 

Increase to accumulated deficit from recording unused MCO credits (204,000) (209,000) 

Accumulated deficit as restated, beginning of year (2,547,096) (2,386,457) 
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3. Significant Accounting Policies 

The financial statements have been prepared in accordance with Canadian generally accepted accounting 
principles established by the Canadian Public Sector Accounting Board. The significant accounting policies are as 
follows: 

(A) ACCRUAL BASIS 

These financial statements are accounted for on an accrual basis whereby expenses are recognized in the fiscal 
year that the events giving rise to the expense occur and resources are consumed. 

(B) VOTED APPROPRIATIONS 

The Office is funded through annual voted appropriations from the Province of Ontario. Unspent appropriations 
are returned to the Province's Consolidated Revenue Fund each year. As the voted appropriation is prepared on a 
modified cash basis, an excess or deficiency of revenue over expenses arises from the application of accrual 
accounting, including the capitalization and amortization of capital assets, the deferral and amortization of lease 
inducements and the recognition of employee benefit costs earned to date but that will be funded from future 
appropriations. 

(C) TANGIBLE CAPITAL ASSETS 

Tangible capital assets are recorded at historical cost less accumulated amortization. Amortization of tangible 
capital assets is recorded on the straight-line method over the estimated useful lives of the assets as follows: 

Computer hardware 3 years 

Computer software 3 years 

Furniture and fixtures 5 years 

Leasehold improvements The remaining term of the lease 

(D) FINANCIAL INSTRUMENTS 

The Office's financial assets and financial liabilities are accounted for as follows: 

• Cash and cash equivalents is subject to an insignificant risk of change in value so carrying value approximates 
fair value. 

• Due from Consolidated Revenue Fund is recorded at cost. 

• Accounts payable and accrued liabilities are recorded at cost. 

• Accrued employee benefits obligation is recorded at cost based on the entitlements earned by employees up to 
March 31, 2012. A fair value estimate based on actuarial assumptions about when these benefits will actually 
be paid has not been made as it is not expected that there would be a significant difference from the recorded 
amount. 

It is management's opinion that the Office is not exposed to any interest rate, currency, liquidity or credit risk 
arising from its financial instruments due to their nature. 
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3. Significant Accounting Policies (Continued) 

(E) MEASUREMENT UNCERTAINTY 

The preparation of financial statements in accordance with Canadian public sector accounting standards requires 
management to make estimates and assumptions that affect the reported amounts of assets and liabilities at the 
date of the financial statements, and the reported amounts of revenue and expenses during the reporting period. 
Items requiring the use of significant estimates include: useful life of capital assets and accrued employee benefits 
obligation. 

Estimates are based on the best information available at the time of preparation of the financial statements and 
are reviewed annually to reflect new information as it becomes available. Measurement uncertainty exists in 
these financial statements. Actual results could differ from these estimates. 



4. Tangible Capital Assets 





Computer 
hardware 
$ 


Computer 
software 
$ 


Furniture Leasehold 
and Fixtures Improvements 
$ $ 


Total 
$ 


Cost 












Opening balance, April 1, 2011 


597,134 


340,833 


378,491 


235,868 


1,552,325 


Additions 


243,826 


49,072 


9,711 


113,955 


416,564 


Disposals 


(153,590) 


(36,919) 


(176,288) 




(366,797) 


Closing balance, March 31, 2012 


687,370 


352,985 


211,914 


349,823 


1,602,092 


Accumulated amortization 












Opening balance, April 1, 2011 


397,966 


147,169 


294,185 


211,101 


1,050,421 


Amortization 


159,472 


100,245 


34,307 


30,465 


324,489 


Disposals 


(153,590) 


(36,919) 


(176,288) 




(366,797) 


Closing balance, March 31,2012 
Net Book Value, March 31,2012 


403,848 


210,495 


152,204 


241,566 


1,008,113 


283,522 


142,490 


59,710 


108,257 


593,979 
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4. Tangible Capital Assets (Continued) 





Computer 


Computer 


Furniture 


Leasehold 


Total 




hardware 


software and Fixtures Improvements 






$ 


$ 


$ 


$ 


$ 


Cost 












Opening balance, April 1, 2010 


585,915 


230,634 


364,666 


235,868 


1,417,083 


Additions 


106,293 


180,559 


13,825 




300,677 


Disposals 


(95,074) 


(70,361) 






(165,435) 


Closing balance, March 31, 2011 


597,134 


340,832 


378,491 


235,868 


1,552,325 


Accumulated amortization 












Opening balance, April 1, 2010 


337,310 


140,165 


237,499 


161,566 


876,540 


Amortization 


155,730 


77,365 


56,686 


49,535 


339,316 


Disposals 


(95,074) 


(70,361) 






(165,435) 


Closing balance, March 31, 2011 


397,966 


147,169 


294,185 


211,101 


1,050,421 














Net Book Value, March 31, 2011 


199,168 


193,663 


84,306 


24,767 


501,904 


5. Accounts Payable and Accrued Liabilities 












March 31,2012 


March 31, 2011 


April 1,2010 








(restated - 


(restated - 








Note 2) 


Note 2) 






$ 


$ 




$ 


Accounts payable 




254,757 


308,990 


351,607 


Accrued salaries and benefits 




409,284 


369,301 


773,293 


Accrued severance, vacation and MCO credits 


983,000 


1,061,000 


1,005,000 






1,647,041 


1,739,291 


2,129,900 



Accounts payable relates largely to normal business transactions with third-party vendors and is subject to 
standard commercial terms. Accrued vacation, salaries and benefits are recorded based on employment 
arrangements and legislated entitlements. 

6. Obligation for Future Employee Benefits 

Although the Office's employees are not members of the Ontario Public Service, under provisions in the Auditor 
General Act, the Office's employees are entitled to the same benefits as Ontario Public Service employees. The 
future liability for benefits earned by the Office's employees is included in the estimated liability for all provincial 
employees that have earned these benefits and is recognized in the Province's consolidated financial statements. 
These benefits are accounted for as follows: 
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6. Obligation for Future Employee Benefits (Continued) 

(A) PENSION BENEFITS 

The Office's employees participate in the Public Service Pension Fund (PSPF) which is a defined benefit pension 
plan for employees of the Province and many provincial agencies. The Province of Ontario, which is the sole 
sponsor of the PSPF, determines the Office's annual payments to the fund. As the sponsor is responsible for 
ensuring that the pension funds are financially viable, any surpluses or unfunded liabilities arising from statutory 
actuarial funding valuations are not assets or obligations of the Office. The Office's required annual payments of 
$719,119 (2011 - $732,873), are included in employee benefits expense in the Statement of Operations and 
Accumulated Deficit. 

(B) ACCRUED EMPLOYEE BENEFITS OBLIGATION 

Although the costs of any legislated severance, compensated absences and unused vacation entitlements earned 
by employees are recognized by the Province when earned by eligible employees, these costs are also recognized 
in these financial statements. These costs for the year amounted to $274,000 (2011 - $225,000) and are included 
in employee benefits in the Statement of Operations and Accumulated Deficit. The total liability for these costs is 
reflected in the accrued employee benefits obligation, less any amounts payable within one year, which are 
included in accounts payable and accrued liabilities, as follows: 





2012 


2011 






(restated - Note 2) 




$ 


$ 


Total liability for severance, vacation and MC0 credits 


3,155,000 


3,049,000 


Less: Due within one year and included in 






accounts payable and accrued liabilities 


983,000 


1,061,000 


Accrued employee benefits obligation 


2,172,000 


1,988,000 



(C) OTHER NON-PENSION POST-EMPLOYMENT BENEFITS 

The cost of other non-pension post-retirement benefits is determined and funded on an ongoing basis by the 
Ontario Ministry of Government Services and accordingly is not included in these financial statements. 

7. Commitments 

The Office has an operating lease to rent premises which expires on October 31, 2021. The minimum rental 
commitment for the remaining term of the lease is as follows: 



$ 

2012- 13 483,000 

2013- 14 488,400 

2014- 15 495,900 

2015- 16 501,300 

2016- 17 508,800 

2017- 18 and beyond 2,412,000 
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8. Public Sector Salary Disclosure Act, 1996 

Section 3(5) of this Act requires disclosure of the salary and benefits paid to all Ontario public-sector employees 
earning an annual salary in excess of $100,000. This disclosure for the 2011 calendar year is as follows: 

Taxable 



Name 


Position 


odidi y 

$ 


Dcllcl llo 

$ 


MfPartpr lim 

IVILfWCtl LCI , Jll 1 1 


Ai irlitnr fnpnprsl 

AAUUILUI VJGI ICI a\ 


?1Q 397 


4 871 


Peall, Gary 


Deputy Auditor General 


167,076 


250 


Bell, Laura 


Director 


114,606 


171 


Bordne, Walter 


Director 


131,000 


196 


Chaqani, Gus 


Director 


114,606 


171 


Chiu, Rudolph 


Director 


131,000 


196 


Fitzmaurice, Gerard 


Director 


131,000 


196 


Gotsis, Vanna 


Director 


106,218 


160 


Klpin Ru^an 

1 \ IC 1 1 1 , v_J U Oka 1 1 


Dirprtor 


131 000 


196 


IVIdZZUHc, VlllUe 


UllcCLUI 


I O I ,uuu 


i yo 


McDowell, John 


Director 


lo 1 ,UUU 


-IOC 

lab 


Pelow, William 


Director 


107,644 


162 


Allan, Walter 


Audit Manager 


105,986 


159 


Amodeo, Paul 


Audit Manager 


100,542 


169 


Carello, Teresa 


Audit Manager 


105,986 


159 


Chan, Sandy 


Audit Manager 


105,986 


159 


Cumbo, Wendy 


Audit Manager 


105,986 


159 


Herberg, Naomi 


Audit Manager 


105,986 


159 


MacNeil, Richard 


Audit Manager 


105,986 


159 


Rogers, Fraser 


Audit Manager 


105,986 


159 


Stavropoulos, Nick 


Audit Manager 


105,986 


159 


Tsikritsis, Emanuel 


Audit Manager 


105,986 


159 


Young, Denise 


Audit Manager 


105,986 


159 


Pedias, Christine 


Manager, Corporate Communications and 


100,658 


145 




Government Advertising Review 






Wiebe, Annemarie 


Manager, Human Resources 


105,986 


159 
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9. Reconciliation to Public Accounts Volume 1 Basis of Presentation 

The Office's Statement of Expenses presented in Volume 1 of the Public Accounts of Ontario was prepared on a 
basis consistent with the accounting policies followed for preparation of the Estimates submitted for approval to 
the Board of Internal Economy, under which purchases of computers and software are expensed in the year of 
acquisition rather than being capitalized and amortized over their useful lives. Volume 1 also excludes the 
accrued employee future benefit costs and deferred lease inducement recognized in these financial statements. A 
reconciliation of total expenses reported in Volume 1 to the total expenses reported in these financial statements 
is as follows: 





2012 


2011 






(restated - 






Note 2) 




$ 


$ 


Total expenses per Public Accounts Volume 1 


15,240,093 


15,002,189 


purchase of capital assets 


(416,564) 


(300,677) 


amortization of capital assets 


324,489 


339,316 


change in accrued future employee benefit costs 


106,000 


122,000 


amortization of deferred lease inducement 


(13,426) 






499 


160,639 


Total expenses per audited financial statements 


15,240,592 


15,162,828 
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10. Expenses by Activity 

2012 





Salaries & 
Benefits 


Other 
Operating 
Expenses 


Statutory 
Expenses 


Total 


% 


Value for money and special audits 
Financial Statement audits 
Pre Election Report 
Government Advertising 


7,417,946 
3,355,738 
143,502 
121,425 


2,233,192 
1,445,314 
46,781 
31,595 


265,415 
24,657 

131,756 
23,271 


9,916,553 
4,825,709 
322,039 
176,291 


65.1 
31.7 
2.1 
1.1 




11,038,611 


3,756,882 


445,099 


15,240,592 


100.0 


% 


72.4 


24.7 
2011 


2.9 


100 






Salaries & 
Benefits 


Other 
Operating 
Expenses 


Statutory 
Expenses 


Total 


% 


Value for money and special audits 
Financial Statement audits 
Pre Election Report 
Government Advertising 


7,837,182 
3,222,451 
33,684 
134,737 


2,076,615 
1,414,654 
6,330 
34,534 


284,827 
22,915 
18,382 
76,517 


10,198,624 
4,660,020 
58,396 
245,788 


67.3 
30.7 
.4 
1.6 




11,228,054 


3,532,133 


402,641 


15,162,828 


100.0 


% 


74.0 


23.3 


2.7 


100 





Expenses have been allocated to the Office's four main activities based primarily on the hours charged to each 
activity as recorded by staff in the Office's time accounting system, including administrative time and overhead 
costs that could not otherwise be identified with a specific activity. Expenses incurred for only one activity, such as 
most travel costs and professional services, are allocated to that activity based on actual billings. 
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11. Deferred Lease Inducement and Receivable 

As part of the lease arrangements for its office premises, the Office negotiated a lease inducement of $322,225 to 
be applied to future accommodation costs. This deferred lease inducement is being amortized as a reduction of 
rent expense on a straight-line basis over the 10-year lease period that commenced November 1, 2011. 

12. Comparative Figures 

Certain comparative figures have been reclassified to conform to the current year's presentation. 

13. Budgeted Figures 

Budgeted figures were approved by the Board of Internal Economy. It is presented for information purposes only 
and has not been audited. 



Exhibit 



Agencies of the Crown 



1. Agencies whose accounts are audited 
by the Auditor General 

Agricorp 

Algonquin Forestry Authority 
Cancer Care Ontario 

Centennial Centre of Science and Technology 
Chief Electoral Officer, Election Finances Act 
Election Fees and Expenses, Election Act 
Financial Services Commission of Ontario 
Grain Financial Protection Board, Funds for 

Producers of Grain Corn, Soybeans, Wheat and 

Canola 

Investor Education Fund, Ontario Securities 

Commission 
Legal Aid Ontario 
Liquor Control Board of Ontario 
Livestock Financial Protection Board, Fund for 

Livestock Producers 
Northern Ontario Heritage Fund Corporation 
Office of the Assembly 
Office of the Children's Lawyer 
Office of the Environmental Commissioner 
Office of the Information and Privacy 

Commissioner 
Office of the Ombudsman 
Ontario Clean Water Agency (December 31)* 
Ontario Development Corporation 
Ontario Educational Communications Authority 
Ontario Electricity Financial Corporation 



Ontario Energy Board 

Ontario Financing Authority 

Ontario Food Terminal Board 

Ontario Heritage Trust 

Ontario Immigrant Investor Corporation 

Ontario Media Development Corporation 

Ontario Mortgage and Housing Corporation 

Ontario Northland Transportation Commission 

Ontario Place Corporation (December 31) * 

Ontario Racing Commission 

Ontario Securities Commission 

Pension Benefits Guarantee Fund, Financial 

Services Commission of Ontario 
Province of Ontario Council for the Arts 
Provincial Advocate for Children and Youth 
Provincial Judges Pension Fund, Provincial Judges 

Pension Board 
Public Guardian and Trustee for the Province of 

Ontario 

2. Agencies whose accounts are audited 
by another auditor under the direction of 
the Auditor General 

Motor Vehicle Accident Claims Fund 
Niagara Parks Commission (October 31) * 
St. Lawrence Parks Commission 
Workplace Safety and Insurance Board 
(December 31)* 



* Dates in parentheses indicate fiscal periods ending on a 
date other than March 31. 




Crown-controlled 
Corporations 



Corporations whose accounts are 
audited by an auditor other than the 
Auditor General, with full access by the 
Auditor General to audit reports, working 



Human Rights Legal Support Centre 
Hydro One Inc. (December 31)* 
Independent Electricity System Operator 

(December 31)* 
McMaster University Foundation 
McMichael Canadian Art Collection 
Metrolinx 

Metropolitan Toronto Convention Centre 
Corporation 

Mississauga Halton Community Care Access Centre 
Mississauga Halton Local Health Integration 
Network 

Municipal Property Assessment Corporation 
North East Community Care Access Centre 
North East Local Health Integration Network 
North Simcoe Muskoka Community Care Access 
Centre 

North Simcoe Muskoka Local Health Integration 
Network 

North West Community Care Access Centre 
North West Local Health Integration Network 
Ontario Agency for Health Protection and 

Promotion 
Ontario Capital Growth Corporation 
Ontario French-language Educational 

Communications Authority 
Ontario Health Quality Council 
Ontario Infrastructure and Lands Corporation 
Ontario Lottery and Gaming Corporation 
Ontario Pension Board (December 31)* 
Ontario Power Authority (December 31)* 
Ontario Power Generation Inc. (December 31)* 



papers and other related documents 

Agricultural Research Institute of Ontario 
Board of Funeral Services 
Brock University Foundation 
Central Community Care Access Centre 
Central East Community Care Access Centre 
Central East Local Health Integration Network 
Central Local Health Integration Network 
Central West Community Care Access Centre 
Central West Local Health Integration Network 
Champlain Community Care Access Centre 
Champlain Local Health Integration Network 
Deposit Insurance Corporation of Ontario 

(December 31)* 
Echo: Improving Women's Health in Ontario 
Education Quality and Accountability Office 
eHealth Ontario 

Erie St. Clair Community Care Access Centre 
Erie St. Clair Local Health Integration Network 
Foundation at Queen's University at Kingston 
Hamilton Niagara Haldimand Brant Community 

Care Access Centre 
Hamilton Niagara Haldimand Brant Local Health 

Integration Network 
HealthForceOntario Marketing and Recruitment 

Agency 

Higher Education Quality Council of Ontario 



* Dates in parentheses indicate fiscal periods ending on a 
date other than March 31. 
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Ontario Tourism Marketing Partnership 

Corporation 
Ontario Trillium Foundation 
Ottawa Convention Centre Corporation 
Owen Sound Transportation Company Limited 
Royal Ontario Museum 
Science North 

South East Community Care Access Centre 
South East Local Health Integration Network 
South West Community Care Access Centre 
South West Local Health Integration Network 
Toronto Central Community Care Access Centre 



Toronto Central Local Health Integration Network 
Toronto Islands Residential Community Trust 

Corporation 
Toronto Waterfront Revitalization Corporation 
Trent University Foundation 
Trillium Gift of Life Network 
University of Ottawa Foundation 
Walkerton Clean Water Centre 
Waterfront Regeneration Trust Agency 
Waterloo Wellington Community Care Access Centre 
Waterloo Wellington Local Health Integration 

Network 




Treasury Board Orders 



Under subsection 12(2) (e) of the Auditor General 
Act, the Auditor General is required to annually 
report all orders of the Treasury Board made to 
authorize payments in excess of appropriations, 
stating the date of each order, the amount author- 
ized and the amount expended. These are outlined 



in the following table. Although ministries may 
track expenditures related to these orders in more 
detail by creating accounts at the sub-vote and item 
level, this schedule summarizes such expenditures 
at the vote and item level. 



1 Ministry 


Date of Order 


Authorized ($) 


Expended ($) 1 


Aboriginal Affairs 


May 19, 2011 


3,000,000 






May 20, 2011 


300,000 


300,000 




Jun 16, 2011 


5,718,600 


5,718,600 




Jun 16, 2011 


286,500 


286,500 




Feb 15, 2012 


21,801,400 


21,797,092 




Mar 22, 2012 


601,000 


406,240 






31,707,500 


28,508,432 


Agriculture, Food and Rural Affairs 


Jun 16, 2011 


11,172,100 






Jul 14, 2011 


4,000,000 






Jul 15, 2011 


800,000 






Aug 11, 2011 


13,402,800 






Aug 17, 2011 


500,000 






Dec 14, 2011 


4,000,000 


4,000,000 




Feb 15, 2012 


3,497,700 






Mar 13, 2012 


11,000,000 


4,875,024 






48,372,600 


8,875,024 


Attorney General 


Apr 14, 2011 


2,700,000 






Jun 16, 2011 


20,100,000 






Mar 22, 2012 


3,139,400 


3,139,400 




Mar 22, 2012 


23,028,600 


17,517,124 






48,968,000 


20,656,524 



Cabinet Office 



Jul 29,2011 



1,000,000 
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Ministry 


Date of Order 


Authorized ($) 


Expended ($) 


Children and Youth Services 


May 19, 2011 


8,190,000 


7,952,253 




Jun 16, 2011 


1,800,000 


- 




Jul 14, 2011 


1,545,000 






Aug 11, 2011 


1,360,000 


— 




Aug 19, 2011 


4,950,800 


— 




Mar 14, 2012 


6,746,500 


— 






24,592,300 


7,952,253 


Citizenship and Immigration 


May 17, 2011 


687,500 


687,500 




Aug 11, 2011 


12,000,000 


8,937,305 




Mar 22, 2012 


266,600 


266,600 




Mar 30, 2012 


2,218,800 


2,182,459 






15,172,900 


12,073,864 


Community and Social Services 


Aug 22, 2011 


642,100 


634,183 




Mar 19, 2012 


4,250,000 


1,967,622 






4,892,100 


2,601,805 


Community Safety and Correctional Services 


Mar 22, 2012 


74,393,700 


71,406,605 




Apr 11, 2012 


3,730,000 


2,705,155 






78,123,700 


74,111,760 


Economic Development and Trade 


Apr 19, 2012 


229,800 


229,800 




Apr 19, 2012 


29,900,000 


17,392,207 






30,129,800 


17,622,007 


Education 


May 19, 2011 


1,710,200 


_ 




Aug 11, 2011 


60,800 


- 




Dec 8, 2011 


51,500 






Mar 19, 2012 


1,556,200 


_ 




Mar 19, 2012 


2,000,000 


_ 




Mar 22, 2012 


1,600,000 


863,214 






6,978,700 


863,214 


Energy 


Aug 9, 2011 


3,874,600 


3,874,600 




Aug 15, 2012 


190,000,000 


166,365,451 






193,874,600 


170,240,051 


Environment 


May 19, 2011 


700,000 


659,962 




Jul 14, 2011 


15,000,000 


5,808,966 




Jul 14, 2011 


3,675,000 


1,419,858 




Mar 22, 2012 


13,965,400 


13,787,865 






33,340,400 


21,676,651 


Finance 


Aug 9, 2011 


3,000,000 






Jan 29, 2012 


3,228,700 


1,234,189 




Feb 13, 2012 


809,900 






Mar 22, 2012 


13,608,100 








20,646,700 


1,234,189 
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Ministry 


Date of Order 


Authorized ($) 


Expended ($) 


Government Services 


Jun 16, 2011 


1,324,400 


670,797 




Jul 14, 2011 


821,400 


821,400 




Aug 9, 2011 


238,000 






Aug 11, 2011 


67,400 






Feb 15, 2012 


550,000 


550,000 




Mar 2, 2012 


1,724,000 


288,662 




Mar 5, 2012 


3,325,200 


3,325,200 






8,050,400 


5,656,059 


Health and Long-Term Care 


Aug 11, 2011 


2,000,000 






Oct 31, 2011 


3,512,800 


3,145,897 




Feb 6, 2012 


8,290,500 


5,824,478 




Feb 15, 2012 


1,791,017,500 


1,744,746,316 




Mar 19, 2012 


1,105,600 






Mar 22, 2012 


90,744,200 


77,509,270 




Apr 19, 2012 


2,000,000 








1,898,670,600 


1,831,225,961 


Health Promotion and Sport 


Aug 9, 2011 


5,000,000 






Aug 9, 2011 


3,000,000 






Mar 19, 2012 


29,629,800 


20,644,554 






37,629,800 


20,644,554 


Infrastructure 


Jun 16, 2011 


36,440,000 






Aug 17, 2011 


52,000,000 


- 




Mar 22, 2012 


20,413,900 






Apr 19, 2012 


2,650,000 








111,503,900 












Labour 


Feb 7, 2012 


752,000 


287,814 


Municipal Affairs and Housing 


Aug 11, 2011 


1,000,000 


1,000,000 




Aug 22, 2011 


5,000,000 


5,000,000 




Mar 22, 2012 


3,015,000 


2,776,101 




Apr 19, 2012 


753,000 


752,707 






9,768,000 


9,528,808 


Natural Resources 


Aug 11, 2011 


2,000,000 


2,000,000 




Aug 11, 2011 


175,000,000 


138,091,753 




Aug 11, 2011 


3,150,000 






Apr 19, 2012 


6,106,400 


6,097,324 






186,256,400 


146,189,077 


Northern Development, Mines and Forestry 


Apr 14, 2011 


13,900,000 


13,900,000 




Jul 14, 2011 


4,500,000 


585,074 




Aug 11,2011 


21,000,000 


12,691,079 




Dec 8, 2011 


792,000 


602,853 




Mar 22, 2012 


450,000 








40,642,000 


27,779,006 



Exhibit 3 ^ 



55 



1 Ministry 


Date of Order 


Authorized ($) 


Expended ($) 










Research and Innovation 


Aug 11, 2011 


45,000,000 


44,997,983 


Revenue 


Nov 22, 2011 


10,000,000 


10,000,000 




Jan 29, 2012 


10,866,000 


9,680,393 




Jan 29, 2012 


34,894,700 


7,168,166 




Feb 13, 2012 


2,182,600 






Feb 15, 2012 


13,682,600 








71,625,900 


26,848,559 


Tourism and Culture 


Jul 14, 2011 


3,500,000 


2,192,105 




Jul 14, 2011 


2,500,000 






Jul 14, 2011 


3,000,000 


3,000,000 




Jul 14, 2011 


4,000,000 






Aug 11, 2011 


400,000 


400,000 




Aug 11, 2011 


2,000,000 


2,000,000 




Aug 11, 2011 


8,000,000 


8,000,000 




Aug 11, 2011 


2,500,000 


2,500,000 




Aug 11, 2011 


2,100,000 


868,562 




Aug 17, 2011 


380,000 


380,000 




Mar 19, 2012 


5,164,800 


5,164,800 




Mar 22, 2012 


200,000 


200,000 




Apr 13, 2012 


8,418,900 


7,872,916 






42,163,700 


32,578,383 


Training, Colleges and Universities 


Jul 14, 2011 


2,000,000 


1,935,317 




Aug 11, 2011 


1,101,700 


1,101,700 




Aug 11, 2011 


8,000,000 






Aug 19, 2011 


6,000,000 


6,000,000 




Dec 8, 2011 


53,000,000 


8,789,742 




Feb 15, 2012 


26,000,000 






Feb 15, 2012 


6,000,000 


6,000,000 




Mar 19, 2012 


451,000 






Mar 22, 2012 


1,984,000 


1,155,210 




Mar 22, 2012 


2,500,000 


2,500,000 




Mar 22, 2012 


2,600,000 






Mar 22, 2012 


500,000 








110,136,700 


27,481,969 


Transportation 


Aug 17, 2011 


8,549,000 






Feb 15 2012 


1,000,000 






Mar 19, 2012 


585,000 






Mar 22, 2012 


6,500,000 


2,400,477 






16,634,000 


2,400,477 



Total Treasury Board Orders 



3,116,632,700 



2,542,034,424 



